Skip to content

Supported & Compatible Data formats

Time Format

The JSON key representing time in the data must have one of the time formats as below. The key representing time and the time format must stay the same for all the messages.

Time Format Example
Seconds 1643707264
Milliseconds 1643707264380
Microseconds 1643707264380000
Nanoseconds 1643707264380000000
ISO 8601 2022-01-31T13:30:00.123456789-07:00

Data Format

MQTT

Data Type Requirements
JSON
  1. Must comply with RFC 8259.
  2. Must have MQTT message size less than 128KB.
  3. Must have a consistent time identifier and format across all the messages.

File Types

Recommendation

Parquet file format is the preferred choice for transferring data via files to TSI. Choosing any other option like CSV or JSON file format can cause file processing delays and is recommended for testing purposes only.

File Type Requirements Examples
Parquet
  1. Must comply with Apache Parquet specifications.
  2. Must be compatible with Apache Spark.
  3. Must not have NULL character in the column name.
  4. Must have an equal number of rows for all the columns.
  5. Must be in wide format if the number of signals in a file exceeds 1000.
  6. Must have a consistent time identifier and format across all files.
  7. Must not have complex types (like lists, structures, or arrays) for column values.
  8. Must have a unique file name.
  9. Should have consistent schema structure across files of the same kind.
  10. It is recommended that row group must not exceed 1,000,000 rows and 10,000 columns. Exceeding this limit may cause processing latency.
 Wide format:
Timestamp,Signal1,Signal2
1643781166702,2.3,5.67

Narrow format:
time,signal,value
1643781166702,signal1,2.3
1643781166702,signal2,5.67
CSV
  1. Must comply with RFC 4180.
  2. Must only have comma (,) as the value separator with whitespaces.
  3. Must not have any summary or metadata information as part of the content.
  4. Must have a consistent time identifier and format across all files.
  5. Must have a unique file name.
  6. Should have consistent schema structure across files of the same kind.
 Wide format:
time,signal1,signal2
1643781166702,2.3,5.67

Narrow format:
time,signal,value
1643781166702,signal1,2.3
1643781166702,signal2,5.67
JSON
  1. Must comply with RFC 8259.
  2. Must have a file size less than 2GB.
  3. Must have a consistent time identifier and format across all files.
  4. Must have a unique file name.
  5. Should have consistent schema structure across files of the same kind.
 Wide format:
[{"time": 1643781166702, "signal1": 2.3, "signal2": 5.67}, {...}]

Narrow format:
[{"time": 1643781166702, "signal": "signal1", "value": 2.3},
{"time": 1643781166702, "signal": "signal2", "value": 5.67}, {...}]
NDJSON
  1. Must comply with RFC 8259.
  2. Must have file extension as .json.
  3. Must have a consistent time identifier and format across all files.
  4. Must have a unique file name.
  5. Must have each line representing a valid JSON object as per RFC 8259.
  6. Should have consistent schema structure across files of the same kind.
 Wide format:
{"time": 1643781166702, "signal1": 2.3, "signal2": 5.67}
{"time": 1643781166703, "signal1": 2.4, "signal2": 5.79}

Narrow format:
{"time": 1643781166702, "signal": "signal1", "value": 2.3}

Other-file-formats

The TSI also accepts compressed file formats. Such files are to be uploaded to a connection via TSI UI only. The above formatted files can be compressed using any one of the acceptable formats: .zip, .tar.gz or .gz.