Is there an industry consensus standard for storing sensor data?

Question

I would like to know what standard(s) may exist for formatting log data from sensors that record information such as:

• temperature
• pressure
• humidity
• vibration
• acceleration vector
• magnetic field vector
• location vector (ex: GPS)
• etc.

I have been an end-user of SCADA software products which record various data fields for each stream of log data it receives into a MySQL database. However, I have never looked into determining how exactly that data was stored. I would like to know if there exists an industry consensus standard for storing sensor data that facilitates information interchange. For example, the Internet Engineering Task Force (IETF) publishes documents called "RFCs" for the purpose of standardizing implementations of commonly used protocols like IMAP (email), TLS (secure web browsing).

For example, such a standard might indicate to me the best practices for:

• formatting units of measurement (ex: degF vs. °F)
• formatting of timestamps of each measurement (ex: ISO-8601)
• file format type (CSV files vs. SQL database)
• delimiters for measurement data (ex: , in .csv files)
• indicating origin and license of data (ex: end device owner/operator, equipment tag number, CC RDFa tags)
• encrypting measurements (esp. in a way compatible with sudden power-loss, ex: DTLS)

While researching this question I noticed that the IETF says the Internet of Things (IoT) is a topic of study for several of its working groups. Its definition of IoT is

The Internet of Things is the network of physical objects or "things" embedded with electronics, software, sensors, actuators, and connectivity to enable objects to exchange data with the manufacturer, operator and/or other connected devices.

However, I did not find any IETF standards document describing standard methods for storing log data. Is there an ISA standard covering methods for logging data? I really like their ISA 5.1 standard for P&ID symbols, "Instrumentation Symbols and Identification".

Thank you for any direction you may provide.

---

Edit-Update (2021-02-02)

After several months of off-and-on research, the closest standard that defines what I was looking for is the Semantic Sensor Network (SSN) Ontology published by the W3C. It is an expanded version of the Sensor, Observation, Sample, and Actuator (SOSA) Ontology. It is capable of associating metadata about an observation (i.e. units, time, location, site-specific names, relationships to other entities in a data collection hierarchy) in an RDF graph data structure. RDF graphs can be "serialized" into JSON (specifically, JSON-LD). The JSON graph data can be formatted using the JSON lines specification so each observation can be appended to a newline-delimited log file in a streaming fashion. Because the logs are machine-readable, an appropriate software package could later import the readings into a database or whatever a user requires.

Search results aren't clear whether SSN will become an industry-standard for general encoding of sensor information but it seems to me the best method for storing sensor data in a self-documenting way. For now, I'm happy to have discovered the method of recording measurements + metadata as an RDF graph using the SSN ontology, serializing the graph as as JSON-LD, and formatting the JSON-LD as JSON Lines that are streamed and appended to a continuously growing log file.

Answer 1

My experience is there is no particular standards for saving sensor data. Usually one just picks a format that is useful to the task at hand. If, for instance one wants to do post processing analysis in Excel then CSV formatted text files work pretty well. If you do use CSV, then just save numerical values. Don't add text indicating units to each number as this would have to be stripped to do any processing. You can provide units in the header row. Commas or tabs are fine as delimiters. Please use the extension ".csv" for comma delimited files. I've seen ".tsv" for tab delimited files. Python and Pandas provide very nice functions for handling CSV files as do many other languages.

I would suggest looking into using the JSON file format. From JSON.org:

JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate.

It is still human readable like CSV, but provides a greater ability to encode useful meta information with the data and more complex data structures. Many, many programming languages have libraries for reading and writing JSON files. I find JSON much less verbose than XML.

As for units, I highly encourage the use of metric units unless there is some particularly customary unit involved.