Audio Anomaly Detection


In this tutorial, we are going to build a predictive maintenance application with an unknown anomaly state for a fan that can run entirely on a cortex-M4 microcontroller using SensiML Analytics Toolkit. These technologies are applicable to a variety of domains where classifying time series sensor data is critical such as gesture recognition, activity recognition, sound classification, and keyword spotting.

Why do we need an unknown anomaly detection?

Anomalies in systems deviate from the standard, happen rarely, and don’t follow the known “pattern”. Depending on your exact use case and application, anomalies only typically occur between 0.001 to 1% of the time — that’s an incredibly small fraction of the data. Therefore, collecting samples for the anomalies is a very expensive process and sometimes it is not possible. On the other hand, failure to detect an anomaly may cause a serious problem.

In this tutorial, we will build a model that will create a prediction for known audio states and return “unknown” for audio anomalies. Our approach is using known states (“OFF”,“ON”) to create an influence area for each class. If a new state falls in the influence area, the model will create a prediction for “OFF” or “ON” states. For any other audio states that fall outside of the influence area, the model will create a prediction of “unknown”.

Area of influence

Area of influence


  1. Demonstrate how to collect and annotate a high-quality dataset of machine state for the fan using the SensiML Data Capture Lab.

  2. Build a data pipeline to extract features in real-time on your target device.

  3. Train a classification model using SensiML custom pipeline.

  4. Convert the model into a Knowledge Pack and flash it to our target embedded device.

  5. Perform live validation of the Knowledge Pack running on-device using the SensiML TestApp.

Capturing Sensor Data

For every machine learning project, the quality of the final product depends on the quality of your curated data set. Time series sensor data, unlike image and audio, are often unique to the application as the combination of sensor placement, sensor type, and event type greatly affects the type of data created. Because of this, you will be unlikely to have a relevant dataset already available, meaning you will need to collect and annotate your own dataset.

To help you to build a data set for your application we have created the SensiML Data Capture Lab, which we are going to use for collecting and annotating data for different fan states. If you are using the demo version of SensiML this project is already uploaded to the account as the Fan State Anomaly Detection From Audio. Otherwise, the dataset for this tutorial can be downloaded at the link below.

  1. Unzip the files and import the dataset to the Data Capture Lab by clicking Import Project

Below you can see a quick demonstration of how the Data Capture lab enabled us to create an annotated fan state data set. In the next few sections, we are going to walk through how we used the Data Capture Lab to collect and label this dataset.

Determining events of interest

Detecting and classifying events is ultimately the main goal of a time series application. In general, events fall into one of two types: continuous or discrete.

Continuous “Status” Events

Continuous events are events that happen over longer, gradual intervals or periods of time. Think of them like you are looking for the current status of the device. An example of this includes a motor fault detection sensor. The sensor will detect a continuous “Normal” status or in a continuous “Failure” status. Another example is a fitness wearable, where you classify the user’s activity status (Running, Walking, Resting).

Why is this important?

The type of event you are trying to detect will change the way you want to train your raw sensor data in the SensiML toolkit. In the SensiML Data Capture Lab, you can put what we call Segment Labels on any portion of your sensor data. This allows you to accurately mark wherein the dataset each type of event is occurring.

Capturing environmental context

In addition to labels, it is also important to capture information about the environment. Capturing this contextual information enables you to build highly tailored models. Additionally, it makes it possible to diagnose why a model might be failing for subsets of your data.

For example, in this fan state dataset, we captured several contextual properties such as data collector, location, surface type, etc. Using these we would be able to build a model for fans that are on different types of surfaces as well as allowing us to understand if certain surfaces perform poorly.

You can capture the contextual information in the Data Capture Lab using the metadata properties. Metadata are applied to the entire captured file, so when you are creating your data collection strategy think carefully about what information you may need. Metadata can be created as a selectable dropdown or manual entry allowing flexibility for your data collectors.

Capturing Data

It’s time to start collecting data. To collect data, we will go to the Capture mode in the Data Capture Lab. The first thing we need to do is to set up the sensor that we would like to use. For this tutorial, we are using the ST SensorTile. There are several other sensors with built-in support. You can see more information about this board here.

We will configure this sensor to capture audio data at a sample rate of 16kHz. In this tutorial, the data is stored directly to an internal SD card then transferred directly to our computer.

After specifying our sensor configuration, we will connect to the device and be ready to record live data. The captured data will be saved locally to your computer as well as be synced up to the SensiML Cloud. This allows other members of your team who have permission to see and label your new captured file. Alternatively, if you already have a data collection method for your device, the Data Capture Lab can import CSV and WAV files directly so you can still use it for annotating the data.

The video below walks through capturing and syncing the data to your cloud project with the Data Capture Lab.

The video below walks through:

  • Open the project

  • Setup sensor configuration for desired edge device

  • Connecting Data Capture Lab to edge device

  • Setup class and metadata labels for data collection

  • Adding data to your cloud project with the Data Capture Lab

Annotating Events of Interest

The Data Capture Lab has a manual label mode and an automatic event detection mode. For this tutorial, we are going to use manual event labels. We will label the regions where the fan is in a particular state. When we go to build the model, we will use a sliding window of 0.25 sec. to generate features to feed into our classifier.

Right-click on the graph to create a segment around your event of interest. Next, click the edit button to select the label that is associated with that event. If you have collected your data set through other means, you can also import CSV or WAV files into the Data Capture Lab. More details can be found here.

The video below walks you through how to label the events of a captured file in the SensiML Data Capture Lab.

Building a Model

The SensiML Analytics Studio is where you can create the following steps - Create a query and pull data into your model - Adding model building steps to the pipeline - Explore the model - Validate model accuracy against raw signal data - Download your model as firmware code for the target device

For the next part of the tutorial, you will need to log into Analytics Studio

Creating a Query

We will first start off by selecting the “Fan State Anomaly Detection From Audio” project which is created with the SensiML Data Capture Lab. When you select the project, you will see an overview of all the captures, queries, pipelines, and models that are part of the project. We will start off by creating a query.

The query is used to select your sensor data from your project. If you need to filter out certain parts of your sensor data based on metadata or labels, you can specify that here.

Prepare Data

Prepare Data

  1. Open the Prepare Data screen to create a query

  2. Create a new query with the following properties:

* Query Name: Q_OFF_ON
* Session: Session_1
* Label: Label
* Metadata: segment_uuid, capture_uuid, DATA
* Source: Select channel_0
* Filter: [Label] IN [OFF,ON]
  1. After clicking Save the Analytics Studio will count all the events of interest you have labeled in this project and graph them

SensiML Python SDK

We are going to connect to SensiML’s cloud engine using the SensiML Python SDK. If you have not yet created an account on SensiML you will need to do that before continuing. You can create a free account by going here

To install the SensiML Python SDK in your environment, run the command in the following cell. !pip install sensiml -U

Import the SensiML Python SDK and use it to connect to SensiML Cloud. Run the following cell, which will ask you for your username and password. After connecting, you will be able to use the SensiML Python SDK to manage the data in your project, create queries, build and test models as well as download firmware. Further documentation for using the SensiML Python SDK can be found here.

from sensiml import *
client = SensiML()

Next we are going to connect to our Fan State Anomaly Detection From Audio project. Run the following cell to connect to the project.

client.project = 'Fan State Anomaly Detection From Audio'

Creating a pipeline

Pipelines are a key component of the SensiML workflow. Pipelines store the preprocessing, feature extraction, and model building steps. When training a model, these steps are executed on the SensiML server. Once the model has been trained, the pipeline is converted to a firmware code that will run on your target embedded device. For more documentation on pipelines see the advanced model building tutorial. To create a new empty pipeline, run the cell below.

client.pipeline = 'Detecting Unknown States'

Now we are going to add our preprocessing steps to the pipeline, for a detailed description of the features available in a pipeline see the documentation here.

You can also list the functions by running client.list_functions(qgrid=False) in a code cell of the notebook.

Adding steps to the pipeline

The pipeline for this tutorial will consist of the following steps:

Input Query

Define the query that will be used in the pipeline.



Signal Transform - Performing Segmentation

The next step is to segment our data into windows which we can perform recognition on. For periodic events, we want to use the Windowing Transform. Delta is the sliding window overlap. Setting delta to the same value as the window size means that there is no overlap in our segmented windows.

Different window sizes can lead to better models. Since We will use MFCC features (it accept the time series <= 400 time points), we will set the window_size and delta to 400.

client.pipeline.add_transform("Windowing", params={"window_size": 400, "delta": 400})

Segment Transform - Strip

This transform shifts the entire signal towards ‘mean’ or ‘min’ before calling the feature generation and classification. In this tutorial we will use the mean options.

    "Strip", params={"input_columns": ["channel_0"], "type": "mean"}

Feature Vector Generation

At this point, we are ready to generate a feature vector from our segments. Feature generators are algorithms to extract relevant feature vectors for model building. In this tutorial we will use MFCC features and set cepstra count to 10. Details of the features can be found here.

Why do we need MFCC features ?

MFCCs are the Mel Frequency Cepstral Coefficients. MFCC takes into account audio perception for sensitivity at appropriate frequencies by converting the conventional frequency to Mel Scale. Therefore, It is quite suitable for audio (frequency based) recognition tasks.

            "name": "MFCC",
            "params": {
                "columns": ["channel_0"],
                "sample_rate": 16000,
                "cepstra_count": 5,

Feature Transform - Isolation Forest Filtering

Removing the outliers and noisy data is useful to improve the robustness of the model. In this tutorial, we will use the Isolation Forest Filtering. List of filters can be found here

client.pipeline.add_transform("Isolation Forest Filtering")

Feature Transform - Feature Cascade

Feature Cascade is a particular case of ensemble learning based on the concatenation of features from consecutive sub segments (cascade). By using all the information collected from each cascade, a model can be created for longer time frames. In “Signal Transform - Performing Segmentation” step, we used segment size of 400 time points (0.025 sec. time frame) which may cause lots of variance with a sample size of 16kHz data. Thus, we need to use longer time frames such as 0.25 sec. One way to achieve that using Feature Cascade with 10 cascades. (10 * 400 = 4000 time points ==> 0.25 sec. time frame)

client.pipeline.add_transform("Feature Cascade", params={"num_cascades": 10, "slide": True})

Feature Transform - Min Max Scale

Our classifiers are optimized for performance and memory usage to fit on resource constrained devices. Because of this we scale the features in the feature vector to be a single byte each so we need to add the Min Max Scale transform to the pipeline. This function will scale the features in the feature vector to have values between 0 and 255.

client.pipeline.add_transform("Min Max Scale")

Classification Algorithm

Classification algorithms take a feature vector as an input and return a classification based on a pre-defined model. The SensiML toolkit supports a variety of classifiers, for this tutorial we will use a classifier with an RBF activation and distance mode of L1.

Why do we need RBF ?

This is a classifier that can create a prediction for “unknown” state. It limits the area of interest with the borders of the known data clusters.

        "classification_mode": "RBF",
        "distance_mode": "L1",

Training Algorithms

The training algorithm attempts to optimize the number of neurons and their locations in order to create the best model. We are using the training algorithm “Hierarchical Clustering with Neuron Optimization,” which uses a clustering algorithm to optimize a neurons placement in the feature space.

Number of neurons may require tuning based on the number of different classes in the data set.

    "Hierarchical Clustering with Neuron Optimization",
    params={"number_of_neurons": 4, "cluster_method": "kmeans"},

Validation Methods

Validation methods are used to check the robustness and accuracy of a model and diagnose if a model is overfitting or underfitting. Since we are trying to create a prediction for unknown states, we want to cover all known states by limiting the influence field with known data. Therefore we will use validation method with a “recall” option. More details regarding the validation methods can be found here

client.pipeline.set_tvo({"validation_seed": 0})

Execute the Pipeline

The final step is where the pipeline is sent to the SensiML Cloud Engine for execution, once the job is completed the results will be returned to you as a model object.

results, stats = client.pipeline.execute()
Executing Pipeline with Steps:

 0.     Name: Q_OFF_ON                              Type: query
 1.     Name: Windowing                             Type: segmenter
 2.     Name: Strip                                 Type: transform
 3.     Name: generator_set                         Type: generatorset
 4.     Name: Isolation Forest Filtering            Type: sampler
 5.     Name: Feature Cascade                       Type: transform
 6.     Name: Min Max Scale                         Type: transform
 7.     Name: tvo                                   Type: tvo
    Classifier: PME
            classification_mode: RBF
            distance_mode: L1
            max_aif: 16384
            min_aif: 2
            num_channels: 1
            reinforcement_learning: False
            reserved_patterns: 0

    Training Algo: Hierarchical Clustering with Neuron Optimization
            aif_method: max
            centroid_calculation: robust
            cluster_method: kmeans
            flip: 1
            linkage_method: average
            max_number_of_weak_vector: 1
            min_number_of_dominant_vector: 3
            number_of_neurons: 4
            singleton_aif: 0

    Validation Method: Recall


Results Retrieved... Execution Time: 0 min. 57 sec.

Model summary

After executing the pipeline, the cloud computes a model summary as well as a confusion matrix. The model summary gives a quick overview of the model performance so we can see what the accuracy of the quantized model was across our data set.

TRAINING ALGORITHM: Hierarchical Clustering with Neuron Optimization

                 F1_SCORE:  100.0  std: 0.00
                PRECISION:  100.0  std: 0.00
              SENSITIVITY:  100.0  std: 0.00



                 F1_SCORE:    train: 100.00  validation: 100.00
              SENSITIVITY:    train: 99.99  validation: 99.99

Confusion matrix

The confusion matrix provides information not only about the accuracy but also what sort of misclassifications occurred. The confusion matrix is often one of the best ways to understand how your model is performing, as you can see which classes are difficult to distinguish between. The confusion matrix here also includes the Sensitivity and Positive Predictivity for each class along with the overall accuracy.

model = results.configurations[0].models[0]
cm = model.confusion_matrix_stats
ClassIn OFF ON UNK UNC Support Sens(%)
0 OFF 7323.0 0.0 0.0 0.0 7323 100.000000
1 ON 0.0 10759.0 2.0 0.0 10761 99.981414
2 UNK 0.0 0.0 0.0 0.0 0 NaN
3 Total 7323.0 10759.0 2.0 0.0 18084 0.000000
4 PosPred(%) 100.0 100.0 NaN 0.0 Acc(%) 99.988940

Saving the Model

Finally, we save the model knowledge pack with a name. This tells the server to persist the model. Models that you persist can be retrieved and viewed in the Analytics Studio in the future. Models that are not saved will be deleted when the pipeline is rerun."Unknown_Anomaly_Detection")
Knowledgepack 'Unknown_Anomaly_Detection' updated.

Exploring the Created Models

In the Explore Model tab, you can get more information about the models that were generated. In this tab you can find the following options:

  • Model Visualization

  • Confusion Matrix

  • Feature Summary

  • Model Summary

  • Pipeline Summary

  • Knowledge Pack Summary

After selecting your pipeline and a model to explore, you can see the performance of your model in confusion matrix that describes how well the model performed at recognizing each class based on validation data. If the selected model is a hierarchical model, the structure of the model and sub-models can be seen in the explore model tab as shown below.

Explore The Model

Explore The Model

Testing a Model in the Analytics Studio

Before you flash the model to the device, you can test the model using the Test Model tab. You can test against any of the captured data files. To do this:

  1. Go to the Explore Model tab of the Analytics Studio.

  2. Select the pipeline you built the model with.

  3. Select the model you want to test.

  4. Select any of the capture files in the Project.

  5. Click RUN to classify that capture using the selected model.

The model will be compiled in the SensiML Cloud and the output of the model will be returned. The graph shows the segment start and segment classified for all the detected events.

Test Model

Test Model

Our model is trained with ‘OFF’ and ‘ON’ audio states. We tested our model with known (‘OFF’, ‘ON’) and unknown states (TAPPING, FAN_GUARDS and BLADE FAULT). Our model is able to create a prediction for the known audio states which are OFF and ON. Also it creates a prediction for all other audio states as “unknown”.


Our model successfully creates a prediction for known and unknown states.

Running a Model On Your Device

Downloading the Knowledge Pack

Now that we have validated our model it is time for a live test. To build the firmware for your specific device go to the Download Model tab of the Analytics Studio. We support the compiled binaries for our target platforms which include fully configured sensors and classification reporting over BLE. We also provide compiled libraries that can be integrated into your application. For enterprise customers, you will have full access to the SDK and can take the compiled models and modify or optimize them for your target devices.

If you are using the community edition of SensiML you can download the firmware binary or library for your device. Head over to the Analytics Studio to download your model and flash it to the device. To download the firmware for this tutorial

  1. Go to the Download Model tab of the Analytics Studio

  2. Select the pipeline and model you want to download

  3. Select the HW platform ST SensorTile

  4. Select Format Binary

  5. To turn on debug output check Advanced Settings and set Debug to True

  6. Click on Output and add BLE as an option as well which enables uart output over serial

  7. Click Download and the model will be compiled and downloaded to your computer.

  8. Unzip the downloaded file and flash to your device.

Download The Model

Download The Model

After downloading the Knowledge Pack, follow the instructions associated with your firmware for flashing it. We have flashing instructions for our supported boards here.

Live test using SensiML TestApp

Being able to rapidly iterate on your model is critical when developing an application that uses machine learning. To facilitate validating in the field, we provide the SensiML TestApp. The TestApp allows you to connect to your microcontroller over Bluetooth and see the classification results live as they are generated by Knowledge Pack running on the microcontroller.

The TestApp also has some nice features, such as the ability to load the class-map, associate images with results, see the history, and apply a majority voting post-processing filter. Documentation on how to use the TestApp can be found here. In this example, we have loaded the TestApp with images of the different machine states to detect. As the machine changes state, the results will be displayed in the TestApp along with the picture and class name.


We hope you enjoyed this tutorial using SensiML Analytics Toolkit. In this tutorial we have covered how to:

  1. Collect and annotate a high-quality data set.

  2. Build a pipeline using SensiML.

  3. Use the SensiML TestApp to perform live validation of the model running on the device.

For more information about SensiML visit our website. To work with us to enable you to build your application get in touch with us.


SensiML enables developers to quickly and easily create machine learning models that run locally on resource-constrained edge devices. SensiML SaaS provides an end to end solution from collecting and annotating a high-quality sensor time-series data set, to analyzing and building data models using AutoML, and finally generating firmware that will run on your target device.