Skip to content

Recommended best practices of using Falkonry TSI

1. Data

Having the right data is essential for effective AI/ML analysis. The quality, relevance, and completeness of input data directly influence the accuracy and usefulness of any insights or detections. Clean, well-structured, and representative data enables models to learn meaningful patterns, while poor data leads to unreliable or biased results. For time series analysis, consistent sampling, sufficient granularity, and proper labeling are especially important. In short, good data is the foundation of good AI. Without it, even the most advanced models cannot deliver actionable intelligence.

Data Preparation

Data format and structure

  • For file-based connections

    • Refer to the File Types and Data Format for
    • Use wide Parquet format to bring production or test data
    • Use wide CSV format to bring test-only data
    • Use unique file names

  • For MQTT-based connections

  • Data should be in JSON format
  • Consistent time identifier, time format, and timezone
  • Signal names in Falkonry TSI are case-sensitive

Live and Historical Data

  • Data older than 4 hours is considered historical
  • Have a dedicated production connection for loading live data for each line
    • Do NOT reuse live connections to upload historical data files
  • A separate file-based historical connection for historical data
    • When processing historical data, use the "Extract Data" action manually

Data Quality and Consistency

  • It is recommended to use 1 Hz
  • Falkonry currently only supports US locale number format (e.g., "12.3" not "12,3").
    • European locale number formats will be considered categorical and dropped from numeric signals
  • Filter out bad data values or quality codes using Custom Rules (JINJA2 scripts)
  • Signals must send data at least once every hour, even if the value doesn't change
  • Avoid sending gappy or quantized data
  • Avoid using custom ETL scripts to build and transfer data to Falkonry.

2. Labeling

When labeling data, especially for semi-supervised models in Falkonry:

  1. Provide more than one label type for multi-class classification
  2. Create different event groups for each label for easier independent modification in iterations
  3. Use the convention of adding _supervised as a prefix to event group names
  4. Provide facts (labels) in smaller time periods that do not overlap on transitions between different conditions or system behaviors to prevent overgeneralization.
  5. For "Early Warning Prediction," when labeling warning periods, consider the desired target warning time, unsupervised model results, and operational cycles (avoiding labeling shutdown periods as warnings).
  6. Ensure normal label periods are at least 10 times longer than warning periods for rare "problem events," and avoid labeling normal periods if there's a possibility of a warning condition present.
  7. Do not use facts to label the "problem events" themselves or other obvious operational abnormalities like shutdown periods, as this can confuse the model's ability to learn "warning" and "normal" states.
  8. If a label isn't appearing in the model condition, ensure the event region is part of the model's learning period, provide enough examples, and confirm they don't overlap in behavior with other events. Adjusting the generalization factor might also help.

3. Learning (or Training) Range

Use known normal time periods of operation to train unsupervised models

Semi-supervised models

  1. Normal label periods should maintain at least a 10:1 ratio to warning periods for rare "problem events"
  2. You should avoid including normal label periods where a warning condition might be present
  3. Falkonry strongly recommends not using "problem events" themselves as facts (labels) in the learning range, as you typically already know when these occurred, and training for them can obscure "warning" and "normal" results.

Impact of Selection

If certain operating behaviors are excluded from the Learn (or training) Range, or if abnormal data is inadvertently included, the system might later capture these behaviors as anomalous when the model goes live.

Refinement

Model refinement often involves adjusting the training periods, which directly relates to the Learning Range. This can be done if the original learning ranges captured insufficient or abnormal data, or if the model output includes "unknown" patterns in validation/test data. Using multiple smaller segments of data for training can also reduce model training time.

4. Signal Management

Having the right set of signals, enriched with sufficient metadata, is a critical first step toward meaningful analysis in Falkonry TSI.

Signal Naming and Metadata

  1. Signals must be uniquely named across Falkonry TSI account
  2. Use an asset-signal form (e.g., asset/signal) to ensure universal uniqueness
  3. Using a prefix (e.g., connection-specific) is recommended for visual distinction
    • Do NOT include "live" or "historical" in the signal prefix.
  4. Have a human-understandable signal name in the signal description
  5. Do NOT change signal names for approved signals once transmitted to Falkonry. Renaming signals should be an exception.
  6. Add signal metadata for more context nad visual treatment on the UI
    • Provide min/max values for all numerical signals

Define Signal Types and Approve

  1. Review draft signals and verify signal type (Numeric or Categorical) before approval
    • Signals like temperature, current, flow, RPM are best used as Numeric for threshold-based rules or AI/ML models
      • If numerical values represent a state, it is recommended to change the data type to Categorical
    • Signals representing system state (e.g., status, alarms, 0s/1s) or textual values (e.g., error codes, batch IDs) are best represented as Categorical and used as "Reference" signals
  2. Approve signals followed by extracting data from the file that brought them in
    • In case the data is coming over of MQTT, no additional action is necessary once the signals are approved

5. Signal Organization

When working with hundreds of signals, navigating an alphabetically ordered list can be inefficient. Falkonry TSI supports flexible signal organization using hierarchies and metadata, making it easier to search, filter, and retrieve relevant signals when needed.

Organize Signals into Trees

  1. Organizing signals into multiple trees
  2. Plan the signal trees, branches, and leaf nodes before assigning signals
  3. It is recommended to assign 30 signals per tree node, not to exceed 50 signals.
  4. Build trees based on:
    • Assets (ISO 14224 recommended) for logical organization.
    • Patterns (identical structure to Assets tree, for /predictions signals).
    • Sensor Types (e.g. temperature, current).
    • Reference (e.g. set points, alarms).
    • Maintenance record (optional, to capture events like delays for RCA).
  5. You can create trees manually or using CSV file import for bulk assignments.
  6. Signal trees are dynamic and can be refined over time as sensor infrastructure evolves.

Signal selection

When selecting signals for use in Falkonry, whether for modeling or rules:

Naming Convention

  1. Signals should be named in an asset-signal format to ensure universal uniqueness
  2. Use a prefix to organize and visually distinguish signals. Maintain consistency with or without the prefix
  3. Avoid escape characters in signal names
  4. Falkonry is case-sensitive; any variation creates a new signal. Approved signal names should generally not be changed; use the signal mapping feature, if necessary.
  5. Implement a naming scheme that helps determine the signal type (e.g., current, voltage) for easier tree building, rule creation, and Insights monitoring
  6. Add signal descriptions as metadata for better understanding within the application

Data Type & Approval

  1. Identify the correct data type (Numeric or Categorical) before processing
  2. Signals like temperature or current are typically Numeric for threshold rules or anomaly detection
  3. Signals representing system state or alarms (e.g., 'open', 'closed', 'true', 'false') are best as Categorical and used as reference signals. Change data type in draft state if mismatched; it cannot be changed after approval.
  4. Approve signals early from the draft list, as data for unapproved signals is not processed

Model/Rule Specific Selection

  1. For rules, all signals must come from a single source (User provided, Insights, Patterns, Rules) and be of the same value type (numeric or categorical)
  2. When using numeric signals for threshold-based rules, include only signals with the exact same value range and applicable threshold criteria; avoid mixing different signal kinds
  3. For categorical signals, ensure they contain categories relevant to the rule criteria
  4. For anomaly score signals (Insights-based), one or more can be selected for different signal types

Signals to Avoid (for Patterns models)

  1. Signals that are flat or monotonous (e.g., counters, durations) over the entire training period
  2. Threshold signals that are flat unless a certain value is exceeded
  3. Highly correlated signals; select only one representative signal from a redundant set
  4. Signals with many gaps, as they can lead to gaps in model output
  5. Signals with vastly different sampling rates (e.g., 1-second and 6-hour data) within the same model; prioritize higher sampling rates
  6. When dealing with data from multiple subsystems, select only those signals relevant to your assessment objective

Performance

  1. Reducing the number of signals in a model will reduce both model training and apply time
  2. Utilize Explanation scores from initial models to identify signals that significantly contribute to conditions of interest, aiding in refining signal selection for subsequent iterations

6. Rules And Alerts

  1. Do NOT include any white space or special characters in the name of the signal. Use an underscore (_) or a hyphen (-) to make the rule name readable.
  2. Use the Rule Description to include information about the possible actions to take when an alert is raised
  3. Use the Mean Rule Statistic to reduce the sensitivity of the rule to transitory spikes

  4. Exclude poorly learned signals from Insights Rule

    When setting up an Insights Rule, exclude any /anomaly score signals that have not been fully learned. These can be identified in the Insights Dashboard by prolonged periods of anomalous behavior. Including such signals too early can result in noisy or unreliable alerts. Wait until Insights has sufficient data to accurately learn their behavior.

  5. Once a Rule has been configured, we recommend monitoring how often the rule is generating True assessments for several days and revising the settings to ensure the rule is generating True assessments and alerts at a rate that matches the criticality of the asset.

7. Model Training Data (Patterns)

Signal Selection for Models

  1. Avoid signals that are flat (single value) over the entire training period
  2. Avoid monotonously increasing or decreasing signals (e.g., counters, lengths, durations, aggregators)
  3. Avoid threshold signals that only change when an underlying value exceeds a threshold
  4. If highly correlated signals exist, select only one representative signal to reduce training time
  5. Avoid signals with many gaps as they tend to produce gaps in model output
  6. Select signals with sampling rates in a well-bounded range (e.g., 1-60 seconds). Avoid signals with extremely low sampling rates (e.g., 6 hours or 1 day alongside 1-second data)
  7. Select only signals relevant to your assessment objective from subsystems
  8. For best results, keep the model signal group under 20 signals
  9. Signal selection as a minimum should have the ability to train different models for the same subsystem with different subsets of signals

Entity Selection (for Multi-entity Models)

  1. When multiple entities are present, use only a few representative entities initially to reduce learning time and prevent overly generalized models
  2. Gradually include more entities as the model generalizes over the remaining ones
  3. All entities must be of the same type with similar behavior and have the same signals and number of signals

Training Period Selection

  1. Divide historical data into distinct training, validation, and test periods
  2. Training data should cover all or most event types/behaviors of interest
  3. Validation data should have at least one or more events of each type
  4. Test the model with data not used for training or validation to prove accuracy and build confidence
  5. The value of training data is measured by the number of events/behaviors/conditions of interest, not just amount or duration
  6. Falkonry recommends at least 200 to 500 batches covering all conditions for a well-trained model
  7. For quicker model iterations with large datasets, use multiple smaller segments of data with behavior of interest rather than one long continuous period

Unsupervised Modeling

  • Start with an unsupervised model as a triage step to get a general assessment of the data and understand distinct behaviors
  • Train unsupervised models exclusively on known healthy data to identify potential unknown faults or anomalies. Avoid using unhealthy data during unsupervised training
  • Use smaller training periods representing multiple known normal modes of behavior
  • Leave the cluster guidance setting at default to allow Falkonry to discover possible distinct behaviors
  • Experiment with a Generalization Factor between 0.3 to 0.5 to control model sensitivity to unusual behaviors.
    • Values > 0.5 reduce unknowns
    • Values < 0.5 are more restrictive
  • The ideal number of conditions detected ranges from 5-8, but can vary

Semi-supervised Modeling

  • Provide more than one label as supervised models work as multi-class classifiers
  • Create different event groups for each label
  • Isolate learn, apply, and failure events in different event groups
  • Label warning periods leading up to "Problem Events" in the training period
  • Label normal period(s) at a ratio of at least 10:1 to warning for rare "Problem Events"
  • Strongly suggest not to use Facts for the Problem Events themselves or any other obvious operational abnormalities, as you generally already know when these occurred, and training for them can obscure warning/normal results
  • Focus first on high detection rate, then on reducing False Warnings (by adding more Normal Events, reducing Warning Events, or reducing Generalization value)
  • To avoid overgeneralization, provide facts in a smaller time period that do not overlap on the transitions between other conditions or system behaviors
  • If 'unknown' labels appear in validation/test data, include that pattern data in the training set or adjust the training period/generalization parameter

Model Evaluation and Output

  • For every model created and evaluated, the system automatically generates an evaluation. Review model evaluations in the iteration cycle.
  • If evaluation periods are greater than 6 months, break them down into 1 to 2 month segments for faster completion
  • Use model output API as it provides continuous data and explanation scores

8. Window Selection

For Sliding Window models, selecting appropriate window bounds is crucial for observing patterns across signals. Here are best practices for window selection:

Review defaults

Do not accept the default window bounds without reviewing them, as they might not be optimal for your specific use case.

Lower bound

  1. It should cover the span of the smallest asset behavior or event of interest.
  2. If this span is unknown, select a time span that includes at least 10 or more sample points of your signal. Fewer points are insufficient for building reasonable feature vectors.

Upper Bound

  1. It should cover the longest event or behavior of your asset.
  2. If unknown, select an upper bound that is 6 to 10 times the size of the lower bound window.

Fixed vs. Optimal Windows

Setting the lower and upper bounds to be the same is possible in some use cases, but it prevents Falkonry Patterns from automatically choosing the optimal time window.

Addressing Unknown Patterns

If an unsupervised model produces unknown patterns in the output, it may indicate that the training data set did not include that pattern or that a lower generalization value is set. In such cases, consider including the unknown pattern data in the training set, adjusting the training period, or increasing the generalization parameter value.

Operational Cycles and Advance Warnings

  1. For "Advance Warning Prediction," the window for recognizing a warning needs to be significantly longer than a single transition time in continuous data, as patterns in a sustained operating mode are more easily distinguished.
  2. The frequency of "problem events" and the targeted advance warning time can provide clues about the phenomena's time constant. For example, a 10-day advance warning might suggest a lower bound of at least 2.5 to 3 hours.

Impact of Window Length

If the chosen observation window is too long, condition recognition can become less accurate and more costly.

Seasonality/Slow Phenomena

For slower-moving phenomena like seasonality, it's often more effective to choose a shorter observation window and provide additional input signals (e.g., day-of-week, external temperature) to capture the longer-term variation or context.

Batch Operations

In cases where the operation structure (like a batch ID) determines the observation window, it can be directly inferred from the operational data

9. General Analysis and Monitoring

Maintain Records for Context:

◦ Capture and load precise ground truth as events for model validation. This includes start/end time, description, action taken, downtime info, issue type, and asset identification.
◦ Capture and load precise and detailed maintenance records as events as soon as maintenance is performed.
◦ Load production schedules as events or signals.

Monitor AI Output Proactively:

◦ Regularly review the Insights dashboard to discover anomalies.
◦ Focus on reviewing findings from the most recent 8 hours.
◦ Prioritize anomalies to analyze: correlated multivariate anomalies and persistent univariate anomalies are most important. Non-obvious correlated short-burst anomalies may be ignored.
◦ Use reports to develop hypotheses, validate events, and document analysis. Start reports in "Personal Reports" and move to "Group Reports" to share.