Connections

In order to bring signal data into the Falkonry Time Series Intelligence for multivariate analysis and real-time monitoring of assets a Connection is required. A Connection is a secure link between TSI and an external data system. A unique Connection is required for each external data system. Many types of external data systems, such as IoT gateways, data historians, SCADA, and OPC systems, can be readily integrated with TSI through various types of supported connections. These connections are designed to work with industry-compliant data infrastructures and operational requirements of the users.

A connection can be monitored from the Connection UI. The Connection UI exposes all the relevant information about the incoming data and the data infrastructure. Each connection has a summary view available in the TSI UI to track the clock time when the recent data was received or sent and the timestamp associated with it.

You need a Owner or Manager role to configure the data connectivity between TSI and external systems. For each data source you must create a connection under the Administration > Connections view. There is no limit on the number of connections.

Inbound connectivity

Outbound connectivity

Supported & Compatible formats

Time Format

The JSON key representing time in the data must have one of the time formats as below. The key representing time and the time format must stay the same for all the messages.

Time Format	Example
Seconds	1643707264
Milliseconds	1643707264380
Microseconds	1643707264380000
Nanoseconds	1643707264380000000
ISO 8601	2022-01-31T13:30:00.123456789-07:00

MQTT Data Format

Data Type	Requirements
JSON	Must comply with RFC 8259. Must have MQTT message size less than 128KB. Must have a consistent time identifier and format across all the messages.

File Types and Data Format

File Type	Requirements	Examples
Parquet	Must comply with Apache Parquet specifications. Must be compatible with Apache Spark. Must not have NULL character in the column name. Must have an equal number of rows for all the columns. Must be in wide format if the number of signals in a file exceeds 1000. Must have a consistent time identifier and format across all files. Must not have complex types (like lists, structures, or arrays) for column values. Must have a unique file name. Should have consistent schema structure across files of the same kind. It is recommended that row group must not exceed 1,000,000 rows and 10,000 columns. Exceeding this limit may cause processing latency.	Wide format: Timestamp,Signal1,Signal2 1643781166702,2.3,5.67 Narrow format: Timestamp,Signal,Value 1643781166702,signal1,2.3
CSV	Must comply with RFC 4180. Must only have comma (,) as the value separator with whitespaces. Must not have any summary or metadata information as part of the content. Must have a consistent time identifier and format across all files. Must have a unique file name. Should have consistent schema structure across files of the same kind.	Wide format: Timestamp,Signal1,Signal2 1643781166702,2.3,5.67 Narrow format: Timestamp,Signal,Value 1643781166702,signal1,2.3
JSON	Must comply with RFC 8259. Must have a file size less than 2GB. Must have a consistent time identifier and format across all files. Must have a unique file name. Should have consistent schema structure across files of the same kind.	Narrow format: [{"Timestamp": 1643781166702, "Signal": "Signal1", "Value": 2.3}, {...}] Wide format: [{"Timestamp": 1643781166702, "Signal1": 2.3, "Signal2": 5.67}, {...}]
NDJSON	Must comply with RFC 8259. Must have file extension as .json. Must have a consistent time identifier and format across all files. Must have a unique file name. Must have each line representing a valid JSON object as per RFC 8259. Should have consistent schema structure across files of the same kind.	Narrow format: {"Timestamp": 1643781166702, "Signal": "Signal1", "Value": 2.3} {"Timestamp": 1643781166702, "Signal": "Signal2", "Value": 5.67} Wide format: {"Timestamp": 1643781166702, "Signal1": 2.3, "Signal2": 5.67}

Note

Parquet file format is the preferred choice for transferring data via files to TSI. Choosing any other option like CSV or JSON file format can cause file processing delays and is recommended for testing purposes only.

The TSI also accepts compressed file formats. Such files are to be uploaded to a connection via TSI UI only. The above formatted files can be compressed using any one of the acceptable formats: .zip, .tar.gz or .gz.