Serial logging: best practice, pitfalls, advice?
Hi, Since there are some really experienced system people here, I'd like to ask some advice on designing logging parameters for a serial device. I'd like to design the output log so it's most obvious to users what's coming out the thing, and so it's easy to stuff readings into a database for later analysis. I have complete freedom to redesign the log coming out of this embedded device. Here's what I get from the device: 1. Approximately twice a minute, I get a cumulative energy reading and the average power for the last 30s or so 2. About once every five minutes, I get a packet which contains an approximate (±1°C or so) ambient temperature I can also report: * a fairly accurate (at least over the short term) elapsed time stamp in milliseconds - but not real time, as there is no RTC. The time stamp rolls over slightly more than once a month. * the received signal strength in dBm This device is decoding the RF packets from the BlueLine Power Cost Meter that the OPA and local utilities give out to hydro consumers in Ontario. It's also the same as the Black & Decker Power Meter. The device would allow you to track real-time energy consumption without fiddling around with Toronto Hydro's time-of-use website (or, gulp, having to mess with Green Button energy files - eek!). As it counts the hydro meter's watt-hour pulses, it's pretty accurate. So - I'm currently logging something like this [105874810] Energy: 63856 Wh, Power: 613 W, Temp: 14 C (Rssi -81 dBm) where the first number's the timestamp. * Would this format cause hiccups in typical logging applications? * Is 9600 baud still the gold standard serial speed? I can use pretty much any speed, and there's never much data to transmit. Any suggestions/questions greatly appreciated. If anyone has one of these OPA Power Cost Meters, and wants to have a play with it, you'll need: * a 3.3 V Arduino (or a standard 5V one and some level converters that can do SPI) * an RFM69 433.92 MHz transceiver board (caution: it uses 2 mm pin spacing, so won't fit a standard breadboard) * this code: https://github.com/CapnBry/Powermon433 The output is rough, but I've successfully tracked several days' power use so far with far higher resolution than is available from smart meter website. cheers, Stewart
On 09/15/2014 11:25 PM, Stewart C. Russell wrote:
Hi,
Since there are some really experienced system people here, I'd like to ask some advice on designing logging parameters for a serial device. I'd like to design the output log so it's most obvious to users what's coming out the thing, and so it's easy to stuff readings into a database for later analysis. I have complete freedom to redesign the log coming out of this embedded device.
Here's what I get from the device:
1. Approximately twice a minute, I get a cumulative energy reading and the average power for the last 30s or so 2. About once every five minutes, I get a packet which contains an approximate (±1°C or so) ambient temperature
I can also report:
* a fairly accurate (at least over the short term) elapsed time stamp in milliseconds - but not real time, as there is no RTC. The time stamp rolls over slightly more than once a month. * the received signal strength in dBm
This device is decoding the RF packets from the BlueLine Power Cost Meter that the OPA and local utilities give out to hydro consumers in Ontario. It's also the same as the Black & Decker Power Meter. The device would allow you to track real-time energy consumption without fiddling around with Toronto Hydro's time-of-use website (or, gulp, having to mess with Green Button energy files - eek!). As it counts the hydro meter's watt-hour pulses, it's pretty accurate.
So - I'm currently logging something like this
[105874810] Energy: 63856 Wh, Power: 613 W, Temp: 14 C (Rssi -81 dBm)
where the first number's the timestamp.
* Would this format cause hiccups in typical logging applications? * Is 9600 baud still the gold standard serial speed? I can use pretty much any speed, and there's never much data to transmit.
Any suggestions/questions greatly appreciated.
If anyone has one of these OPA Power Cost Meters, and wants to have a play with it, you'll need:
* a 3.3 V Arduino (or a standard 5V one and some level converters that can do SPI) * an RFM69 433.92 MHz transceiver board (caution: it uses 2 mm pin spacing, so won't fit a standard breadboard) * this code: https://github.com/CapnBry/Powermon433
The output is rough, but I've successfully tracked several days' power use so far with far higher resolution than is available from smart meter website.
cheers, Stewart
It's just a little redundant: the names end in : the values in , the time is in square brackets, the rssi in round ones. To analyze, I need to first convert the closing bracket to a , and the opening to a blank, and the reverse with the round brackets. Then I prune off the keywords, : and units. At that point I can start the analysis. I'd be more inclined toward Timestamp: 105874810, Energy (Wh): 63856, Power (W): 613, Temp (C): 14, Rssi (dBm): -81 or a linear format like Timestamp, Energy (Wh), Power (W), Temp (C), Rssi (dBm) 105874810, 63856, 613, 14, -81 If the arduino has a clock, you might also start with *# Date and time, **Tue Sep 16 14:43:01 EDT 2014* or if the clock was fairly good at start time, compute a date and time for each line from the millisecond counter and the start time. --dave -- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
Thanks to all who responded! On 14-09-16 02:45 PM, David Collier-Brown wrote:
It's just a little redundant: the names end in : the values in , the time is in square brackets, the rssi in round ones.
Good point. I wonder if it's worth encoding each record as JSON, like: {"Energy_Wh": 63856, "Temp_C": 14, "Rssi_dBm": -81, "Power_W": 613, "Timestamp_ms": 105874810} Readable, computer readable; maybe a little verbose, but definitely self-describing.
If the arduino has a clock, you might also start with *# Date and time, **Tue Sep 16 14:43:01 EDT 2014*
It doesn't, unfortunately; also, many Arduinos use ceramic resonators, so their clock stability is pretty terrible. I'll have to assume that the computer it's talking to has a decent RTC, and can use the system time. === On 14-09-16 03:04 PM, D. Hugh Redelmeier wrote:
I'd be tempted to put some kind of checksum on each record. Communications has many ways of screwing up.
We're decoding someone else's protocol. The protocol, which a couple of folks smarter than me reverse-engineered, does use CRC8 in its packets, and uses multiple transmissions to get through. The transmit hardware is quite nicely engineered: sealed case, decent Atmel transmitter hardware with a proper loop antenna. The stock receiver uses those horrible cheap OOK RF Radio modules, but works. The RFM69 transceiver module I'm using as a receiver is rather nice; built-in packet protocols, a decent on-chip buffer; even on-chip AES encryption/decryption, should you need it.
I'd encode it in ASCII graphics somehow (or even decimal) so that your format remains readable.
Oh I wish the developer had done this, but most ISM band transmissions have to be super short, as the transmitter duty cycle is really low. Plus, every single car remote, baby alarm and automatic light switch is chirping away on the same frequency. I'm amazed any data gets through at all! cheers, Stewart
On 09/17/2014 08:51 AM, Stewart C. Russell wrote:
Thanks to all who responded!
On 14-09-16 02:45 PM, David Collier-Brown wrote:
It's just a little redundant: the names end in : the values in , the time is in square brackets, the rssi in round ones. Good point. I wonder if it's worth encoding each record as JSON, like:
{"Energy_Wh": 63856, "Temp_C": 14, "Rssi_dBm": -81, "Power_W": 613, "Timestamp_ms": 105874810}
Readable, computer readable; maybe a little verbose, but definitely self-describing.
I'd says yes if you're going to analyze with something other than awk. --dave (awk: squak! thud...) c-b -- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
On Wed, Sep 17, 2014 at 10:43 AM, David Collier-Brown <davec-b@rogers.com> wrote:
I'd says yes if you're going to analyze with something other than awk.
I use jq[1] as an awk like parser for JSON on the command line. [1]: http://stedolan.github.io/jq/
Goes perfectly with pup for rest calls that return json|xml --dave On 09/17/2014 02:01 PM, Myles Braithwaite wrote:
On Wed, Sep 17, 2014 at 10:43 AM, David Collier-Brown <davec-b@rogers.com> wrote:
I'd says yes if you're going to analyze with something other than awk. I use jq[1] as an awk like parser for JSON on the command line.
[1]: http://stedolan.github.io/jq/
--- GTALUG Talk Mailing List - talk@gtalug.org http://gtalug.org/mailman/listinfo/talk
-- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
Are people getting multiple copies of talk messages? I get two (plus any BCC for ones I send) --dave -- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
| From: Stewart C. Russell <scruss@gmail.com> | Any suggestions/questions greatly appreciated. I'd be tempted to put some kind of checksum on each record. Communications has many ways of screwing up. I'd encode it in ASCII graphics somehow (or even decimal) so that your format remains readable.
I would make the logging as easy for a computer program to parse while still making it readable. Bill On Mon, Sep 15, 2014 at 11:25 PM, Stewart C. Russell <scruss@gmail.com> wrote:
Hi,
Since there are some really experienced system people here, I'd like to ask some advice on designing logging parameters for a serial device. I'd like to design the output log so it's most obvious to users what's coming out the thing, and so it's easy to stuff readings into a database for later analysis. I have complete freedom to redesign the log coming out of this embedded device.
Here's what I get from the device:
1. Approximately twice a minute, I get a cumulative energy reading and the average power for the last 30s or so 2. About once every five minutes, I get a packet which contains an approximate (±1°C or so) ambient temperature
I can also report:
- a fairly accurate (at least over the short term) elapsed time stamp in milliseconds - but not real time, as there is no RTC. The time stamp rolls over slightly more than once a month. - the received signal strength in dBm
This device is decoding the RF packets from the BlueLine Power Cost Meter that the OPA and local utilities give out to hydro consumers in Ontario. It's also the same as the Black & Decker Power Meter. The device would allow you to track real-time energy consumption without fiddling around with Toronto Hydro's time-of-use website (or, gulp, having to mess with Green Button energy files - eek!). As it counts the hydro meter's watt-hour pulses, it's pretty accurate.
So - I'm currently logging something like this
[105874810] Energy: 63856 Wh, Power: 613 W, Temp: 14 C (Rssi -81 dBm)
where the first number's the timestamp.
- Would this format cause hiccups in typical logging applications? - Is 9600 baud still the gold standard serial speed? I can use pretty much any speed, and there's never much data to transmit.
Any suggestions/questions greatly appreciated. If anyone has one of these OPA Power Cost Meters, and wants to have a play with it, you'll need:
- a 3.3 V Arduino (or a standard 5V one and some level converters that can do SPI) - an RFM69 433.92 MHz transceiver board (caution: it uses 2 mm pin spacing, so won't fit a standard breadboard) - this code: https://github.com/CapnBry/Powermon433
The output is rough, but I've successfully tracked several days' power use so far with far higher resolution than is available from smart meter website.
cheers, Stewart
--- GTALUG Talk Mailing List - talk@gtalug.org http://gtalug.org/mailman/listinfo/talk
participants (5)
-
Bill Thanis -
D. Hugh Redelmeier -
David Collier-Brown -
Myles Braithwaite -
Stewart C. Russell