Adaptive downsampling of time-series data?

Question

I’ve been tasked with developing software to (among other things) log temperature data from samples in a furnace. Generally the temperatures are very steady, but can, at times, experience dramatic changes.

I’ll be storing the data in SQLite, which is more than adequate for millions of rows, but this will be running on something like a Raspberry Pi with 8 GB of RAM. I’m doing testing now, but I’d rather not store data I don’t need to store.

The data should be easy to downsample regions of data without going below Nyquist, but when I search for “adaptive decimation” or “adaptive sampling,” I get some pretty weighty scholarly articles. I would have thought this wasn’t so hard to do, and widely available. I can do it after the fact, I don’t need to predict what my data is going to do. I don’t mind periodically going over older data to do the decimation.

It’s possible Pandas or some other Python library can do it with a simple call, so that might be a starting point for me (half the battle is figuring out what the technique is called). I’m writing this in Swift, so I don’t expect a ready-made solution, but if I knew what to implement, I could.

Any suggestions on how to accomplish this?

Answer 1

A technique I've seen used for air conditioning units, is that you can create a new log when a) the temperature stops being constant or b) changes in slope.

In essence, if for a long time you have constant temperature you record nothing.

When you start seeing departures, you log the last known point of constant temperature.

then you have two paths:

• either record at a set interval until the temperature remains constant.
• or monitor the trend, and as long as it is linear within a margin of error you do nothing. If it deviates, then you record the last known step and move on.

The technique above (depending on your system and sampling settings) can have a significant savings.

You need to keep in mind that it can get tricky because of random fluctuations in the sensor (you need to know the noise of the sensor and the environment).

Another thing, is that if you are monitoring multiple sensors, then you won't take measurements at the same point in time. Sometimes, that might cause an extra hassle when you are trying to analyze data.