Data Logging

This page describes the data logging process and structure of the sensor node.

Note

Data logging and memory management were my absolute weakest link in this project, and for this reason I spent a lot of time trying to understand data logging and memory concepts and not as much time actually putting them into practice and trying things as I would have liked. The next person to work on this project will find that there is little incorporated into the code to achieve data logging, and little direction in the comments.

In this Wiki, I give an overview of the original vision and some places to start, however in the hands of someone with more experience in memory management, there could be an obvious approach that I have not tried. In this case, I encourage you to try it.

Logging Strategy

The accelerometer can collect data very quickly, store it in a register, and perform functions like trigger interrupts. The microphone is both slow and stupid, such that it will not tell you when the data was taken or save it - all of the useful functionality comes from the Silicon Labs PDM driver.

[—! include a note about how you calculated that value —]

Therefore the strategy is the following:

  1. In the app_init() function, a permanent array for data will need to be created. For one second of data, this will be about 132kB. For clarity, I will refer to this location as the data array for the rest of this list.
  2. Receive the go-ahead from the gateway. This will likely be in the form of a command or a write to a user-defined register; in this case, this entire data collection and logging method should be put into the Event Handler on the Bluetooth end of things. In this scenario, the final product will have no code at all under app_process_action in app.cpp; only in app_init() and in sl_bt_on_event(sl_bt_msg_t *evt).
  3. Initiate the timer module and start the timer as soon as the go-ahead has been received. I strongly recommend the TIMER method, instead of BUTCC, RTCC or LETIMER - that is because TIMER can use the fastest clock for its counter, and therefore we will have the most precision in our timestamps.
  4. Start the timer.
  5. Take readings of temperature and humidity at the top of the collection period and store them in the data array at the first free location. Ensure that these are timestamps on both readings as close as when they happened as possible.
  6. At this point, you may need to initiate a SECOND timer to keep track of the time period. You could also assign the data array to be a fixed length that you calculate either in the code or by yourself and then implement for the sake of testing.
  7. Begin the accelerometer and the mic interrupts.
    1. In each mic interrupt, store the PDM data with one timestamp each in the next free spaces in the data array.
    2. In each accelerometer interrupt, make note of the time stamp, read in all the data from the accelerometer buffer, and assign an impossible timestamp to all but the last entry (0x00 would be a great choice), which will then be the saved timestamp from the top of the interrupt event. In this way, we can go back and fix the data after.
  8. When the data collection period has stopped (either from the second timer or from the data array filling up), disable both mic and accelerometer interrupts.
  9. Take last data readings for the temperature sensor and the hygrometer sensor, with timestamps.
  10. Disable the timers.
  11. Write to the GATT that the data is ready to be read from the gateway and await the response.
  12. The sensor will then start sending indications to the gateway that will send all the data from the data array through Bluetooth (to be processed and saved to a file on the gateway side). This data can only be sent over 512B at at time. When the confirmation has been received, it will move up to the next section of data and continue until the entire array has been transferred over.
  13. When all the data is transferred, empty the data array. This must be done in page-length increments, which is 8kB for our device.

Notes:

  • The gateway sends the NST time to the sensor, but the sensor does not interact with it. This is just to ensure that we can re-sync the timestamps when we send them back to the gateway. It will waste time if we try and convert each counter value to the real time.
  • For every point of data (be it 1B or 2B data), it will need to be followed by a 1b-bit counter value as the timestamp and then a final byte that I will call a "label" to designate the type of data that is being saved, to distinguish between the sensors. This label should be as hard as possible to mistake for a sensor reading or a timestamp.

Storage Options

There are several ways of saving data to this chip, including some that are well-documented for other SiLabs chips [1].

  • em_msc (emlib)
  • NVM3 Driver (emdrv + emlib) [2] [3]
  • EEPROM (eeprom + emlib)

See Also

Bibliography
1. Silicon Labs Community Forum; 2016; efm32gg data storage https://www.silabs.com/community/mcu/32-bit/forum.topic.html/efm32gg_data_storage-8zk0
2. Silicon Labs Community Forum; 2020; Using nvm3 for user data; https://www.silabs.com/community/wireless/bluetooth/forum.topic.html/using_nvm3_for_userdata-eLCv
3. Silicon Labs; AN1135: Using Third Generation NonVolatile Memory (NVM3) Data Storage; https://www.silabs.com/documents/public/application-notes/an1135-using-third-generation-nonvolatile-memory.pdf

Potential References

These links are references that I have found but have not had time to investigate in depth.

*

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License