Skip to content

File Connection with Private S3 Settings

A Connection in Falkonry Time Series Intelligence is a virtual link between TSI and the external data system. A unique Connection is required to connect each of the external data systems in order to bring signal data into TSI for analysis and real-time monitoring of the assets. The external data systems such as IoT gateways, data historians, SCADA, and OPC systems can be readily integrated with AI Cloud through various types of Connections offered by TSI. One of the Inbound Connection types is :doc: [File Connection \<cloudbase/InboundConnections>]{.title-ref}.

The File Connection uses the AWS S3 bucket as a way to transfer data files into TSI. As files are transferred into the S3 bucket, AI Cloud starts processing the files in order of their arrival. This S3 bucket could either be hosted in Falkonry's or the User's AWS infrastructure.

When using a User-managed S3 bucket, a file-connector is installed in the User's AWS account as an AWS lambda function. This file connector is responsible for communicating the arrival of new data files in the targeted S3 bucket to AI Cloud. The connector uses Lambda \<> S3 binding managed by AWS and uses token-based authentication to communicate with AI Cloud over port 443 (HTTPS). On the other side of the integration, AI Cloud requires AWS credentials to be able to read these files from the User managed S3 bucket. These AWS credentials are required during the Connection creation process.

The Falkonry-FileConnector-SQS SQS triggers the Falkonry-FileConnector lambda. The Falkonry-FileConnector lambda notifies AI Cloud to process the files. TSI reads the data files from the S3 bucket using the IAM credentials configured for this Connection.

image

Creating a Secure S3 Bucket in AWS

  1. Create a new S3 bucket from your AWS console. Following are recommended settings for the new bucket.
  1. ACLs must be disabled. This is to enforce the IAM policy on all the files within this S3 bucket.
  2. Public access must be blocked. This is to block all the public access to the files within this S3 bucket.
  3. All AWS regions are supported.
  4. Bucket versioning is optional. AI Cloud expects all the files within a folder to have unique names. And the files once written should never be modified.
  5. Encryption is optional. AI Cloud can be integrated with this S3 bucket to read server-side encrypted or unencrypted files. It is recommended to turn ON the server-side encryption.
  6. Object lock must be disabled. There is no requirement for the files within the S3 bucket to be locked.
  1. Create a folder named data inside this new S3 bucket.
3. Create a new IAM user with programmatic access and attach the following policy.

json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::example-bucket-name", "arn:aws:s3:::example-bucket-name/*" ] } ] }

  1. Generate an access key for this new IAM user from the AWS console.

Creating a File Connection

  1. Go to UI and navigate to the New Connection view.
  2. Use the S3 Bucket name, AWS Region, Access Key, and Access Secret as created in the above section.
  1. Choose the "Use Custom S3 Bucket" option
  2. Set S3 Bucket Path as s3://example-bucket-name/data
  3. Set AWS Region as the region selected while creating the S3 bucket.
  4. Set AWS Access Key as the Access Key generated for the new IAM user.

. Set AWS Secret Key as the Access Secret generated for the new IAM user.

image
  1. The AWS credentials get verified when creating the Connection.

Installing File Connector in AWS

  1. Get the link to download the File Connector package zip from Falkonry Team.
  2. Extract the downloaded zip and validate that it contains the folder named lambda and cloudformation_template.yml file.
  3. Upload the falkonry-fileconnector-package folder to the newly created S3 bucket at the root level.
  4. Go to AWS console and navigate to the CloudFormation section and click "Create stack (with new resources)" option.
5. Select the "Template is ready" option and upload the cloudformation_template.yml file from the downloaded zip.

image

  1. Give a Stack Name and provide the parameters on the next screen.
  1. Set AuthorizationToken as API Token generated from AI Cloud UI under Administration section.
  2. Set ConnectionId as the ID of the newly created Connection.
  3. Set DataBucketName as the name of the newly created S3 bucket.
  4. Set TenantId as the account ID in AI Cloud.
  5. Set lambdaBucketName as the name of the newly created S3 bucket.
  1. Click "Next" and leave the default settings as it is.
  2. Click "Create Stack" to begin the File Connector installation.
  3. Once the Stack creation completes, validate that the AWS Lambda function exists with the name Falkonry-FileConnector and AWS SQS queue exists with name Falkonry-FileConnector-SQS.
  4. Navigate to the newly created S3 bucket and open Properties > Event notifications section.
  5. Create a new Event Notification using the following settings.
  1. Give Event a name.
  2. Select "All object create" events under Event Types.
  3. Select Destination as SQS queue and select SQS queue with name Falkonry-FileConnector-SQS.

After Installation

Users can now upload data files to the data folder inside the newly created S3 bucket.