Training Algorithms
Training algorithms are used to select the optimal Parameters of a model.

Hierarchical Clustering with Neuron Optimization
Hierarchical Clustering with Neuron Optimization takes as input feature vectors, corresponding class labels, and desired number of patterns, and outputs a model.
Each pattern in a model consists of a centroid, its class label, and its area of influence (AIF). Each centroid is calculated as an average of objects in the cluster, each class label is the label of the majority class, and each AIF is the distance between the centroid and the farthest object in that cluster.
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
number_of_neurons (int) – the maximum number of output clusters (neurons) desired
linkage_method (str) – options are average, complete, ward, and single (default is average)
centroid_calculation (str) – options are robust, mean, and median (default is robust)
flip (int) – default is 1
cluster_method (str) – options are DLCH, DHC, and kmeans (default is DLHC)
aif_method (str) – options are min, max, robust, mean, median (default is max)
singleton_aif (int) – default is 0
min_number_of_dominant_vector (int) – It is used for pruning. It defines min. number of vector for dominant class in the cluster.
max_number_of_weak_vector (int) – It is used for pruning. It defines max. number of vector for weak class in the cluster.
 Returns
one or more models

Load Model PME
Load Neuron Array takes an input of feature vectors, corresponding class labels, and a neuron array to use for classification. The neuron array is loaded and classification is performed.
Note: This training algorithm does not perform optimizations on the provided neurons.
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
neuron_array (list) – A list of neurons to load into the hardware simulator.
class_map (dict) – class map for converting labels to neuron categories.
 Returns
a set of models

Load Model TensorFlow Lite for Microcontrollers
Provides the ability to upload a TensorFlow Lite flatbuffer to use as the final classifier step in a pipeline.
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
model_parameters (int) – The flatbuffer object of your TensorFlow micro model
class_map (dict) – class map for converting labels to output
estimator_type (str) – defines if this estimator performs regression or classification, defaults to classification
threshold (float) – if no values are greater than the threshold, classify as Unknown
train_history (dict) – training history for this model
model_json (dict) – expects the model json file from the tensorflow api tf_model.to_json()
Example
SensiML provides the ability to train and bring your own NN architecture to use as the classifier for your pipeline. This example starts from the point where you have created features using the SensiML Toolkit
>>> x_train, x_test, x_validate, y_train, y_test, y_validate, class_map = >>> client.pipeline.features_to_tensor(fv_t, test=0.0, validate=.1)
Tensorflow Lite Micro only supports a subset of the full tensorflow functions. For a full list of available functions see the all_ops_resolver.cc. Use the Keras tensorflow API to create the NN graph.
>>> from tensorflow.keras import layers >>> import tensorflow as tf >>> tf_model = tf.keras.Sequential() >>> tf_model.add(layers.Dense(12, activation='relu',kernel_regularizer='l1', input_shape=(x_train.shape[1],))) >>> tf_model.add(layers.Dropout(0.1)) >>> tf_model.add(layers.Dense(8, activation='relu', input_shape=(x_train.shape[1],))) >>> tf_model.add(layers.Dropout(0.1)) >>> tf_model.add(layers.Dense(y_train.shape[1], activation='softmax')) >>> tf_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) >>> tf_model.summary() >>> train_history = {'loss':[], 'val_loss':[], 'accuracy':[], 'val_accuracy':[]}
Train the Tensorflow Model
>>> epochs=100 >>> batch_size=32 >>> data = tf.data.Dataset.from_tensor_slices((x_train, y_train)) >>> shuffle_ds = data.shuffle(buffer_size=x_train.shape[0], reshuffle_each_iteration=True).batch(batch_size) >>> history = tf_model.fit( shuffle_ds, epochs=epochs, batch_size=batch_size, validation_data=(x_validate, y_validate), verbose=0) >>> for key in train_history: >>> train_history[key].extend(history.history[key]) >>> import sensiml.tensorflow.utils as sml_tf >>> sml_tf.plot_training_results(tf_model, train_history, x_train, y_train, x_validate, y_validate)
Qunatize the Tensorflow Model
The
`representative_dataset_generator()`
function is necessary to provide statistical information about your dataset in order to quantize the model weights appropriatley.The TFLiteConverter is used to convert a tensorflow model into a TensorFlow Lite model. The TensorFlow Lite model is stored as a flatbuffer which allows us to easily store and access it on embedded systems.
Quantizing the model allows TensorFlow Lite micro to take advantage of specialized instructions on cortexM class processors using the cmsisnn DSP library which gives another huge boost in performance.
Quantizing the model can reduce size by up to 4x as 4 byte floats are converted to 1 byte ints in a number of places within the model.
>>> import numpy as np >>> def representative_dataset_generator(): >>> for value in x_validate: >>> # Each scalar value must be inside of a 2D array that is wrapped in a list >>> yield [np.array(value, dtype=np.float32, ndmin=2)] >>> >>> converter = tf.lite.TFLiteConverter.from_keras_model(tf_model) >>> converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE] >>> converter.representative_dataset = representative_dataset_generator >>> tflite_model_quant = converter.convert()
Uploading Trained TF Lite model to SensiML
>>> class_map_tmp = {k:v+1 for k,v in class_map.items()} #increment by 1 as 0 corresponds to unknown >>> client.pipeline.set_training_algorithm("Load Model TensorFlow Lite for Microcontrollers", >>> params={"model_parameters": { >>> 'tflite': sml_tf.convert_tf_lite(tflite_model_quant)}, >>> "class_map": class_map_tmp, >>> "estimator_type": "classification", >>> "threshold": 0.0, >>> "train_history":train_history, >>> "model_json": tf_model.to_json() >>> }) >>> client.pipeline.set_validation_method("Recall", params={}) >>> client.pipeline.set_classifier("TensorFlow Lite for Microcontrollers", params={}) >>> client.pipeline.set_tvo() >>> results, stats = client.pipeline.execute() >>> >>> results.summarize()

Neuron Optimization
Neuron Optimization performs an optimized grid search over KNN/RBF and the number of neurons for the parameters of Hierarchical Clustering with Neuron Optimization. Takes as input feature vectors, corresponding class labels and outputs a model.
Each pattern in a model consists of a centroid, its class label, and its area of influence (AIF). Each centroid is calculated as an average of objects in the cluster, each class label is the label of the majority class, and each AIF is the distance between the centroid and the farthest object in that cluster.
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
neuron_range (list) – the range of max neurons spaces to search over specified as [Min, Max]
linkage_method (str) – options are average, complete, ward, and single (default is average)
centroid_calculation (str) – options are robust, mean, and median (default is robust)
flip (int) – default is 1
cluster_method (str) – options are DLCH, DHC, and kmeans (default is DLHC)
aif_method (str) – options are min, max, robust, mean, median (default is max)
singleton_aif (int) – default is 0
min_number_of_dominant_vector (int) – It is used for pruning. It defines min. number of vector for dominant class in the cluster.
max_number_of_weak_vector (int) – It is used for pruning. It defines max. number of vector for weak class in the cluster.
 Returns
one or more models
“description”: “.”}
return None

Bonsai Tree Optimizer
Train a Bonsais Tree Classifier using backpropagation.
For detailed information see the ICML 2017 Paper
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
epochs (str) – The number of training epochs to iterate over
batch_size (float) – The size of batches to use during training
learning_rate (float) – The learning rate for training optimization
project_dimensions (int) – The number of dimensions to project the input feature space into
sigma (float) – tunable hyperparameter
reg_W (float) – regularization for W matrix
reg_V (float) – regularization for V matrix
reg_Theta (float) – regularization for Theta matrix
reg_Z (float) – regularization for Z matrix
sparse_V (float) – sparcity factor for V matrix
sparse_Theta (float) – sparcity factor for Theta matrix
sparse_W (float) – sparcity factor for W matrix
sparse_Z (float) – sparcity factor fo Z matrix
 Returns
model parameters for a bonsai tree classifier

Train Fully Connected Neural Network
Provides the ability to train a fully connected neural network model to use as the final classifier step in a pipeline.
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
class_map (dict) – optional, class map for converting labels to output
estimator_type (str) – defines if this estimator performs regression or classification, defaults to classification
threshold (float) – if no values are greater than the threshold, classify as Unknown
dense_layers (list) – The size of each dense layer
drop_out (float) – The amount of dropout to use after each dense layer
batch_normalization (bool) – Use batch normalization
final_activation (str) – the final activation to use
iteration (int) – Maximum optimization attempt
batch_size (int) – The batch size to use during training
metrics (str) – the metric to use for reporting results
learning_rate (float) – The learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.
batch_size – Refers to the number of training examples utilized in one iteration.
loss_function (str) – It is a function that determine how far the predicted values deviate from the actual values in the training data.
tensorflow_optimizer (str) – Optimization algorithms that is used to minimize loss function.
Example
SensiML provides the ability to train NN architecture to use as the classifier for your pipeline. Tensorflow Lite Micro only supports a subset of the full tensorflow functions. For a full list of available functions see the all_ops_resolver.cc. Use the Keras tensorflow API to create the NN graph.
>>> client.project = 'Activity_Detection' >>> client.pipeline = 'tf_p1'
>>> client.pipeline.stop_pipeline()
>>> sensors = ['GyroscopeX', 'GyroscopeY', 'GyroscopeZ', 'AccelerometerX', 'AccelerometerY', 'AccelerometerZ']
>>> client.pipeline.reset()
>>> client.pipeline.set_input_query("Q1")
>>> client.pipeline.add_transform("Windowing", params={"window_size":200, "delta":200, "train_delta":0})
>>> client.pipeline.add_feature_generator([ {'name':'MFCC', 'params':{"columns":sensors,"sample_rate":100, "cepstra_count":1}} ])
>>> client.pipeline.add_transform("Min Max Scale")
>>> client.pipeline.set_validation_method("Recall", params={})
>>> client.pipeline.set_training_algorithm("Train Fully Connected Neural Network", params={ "estimator_type":"classification", "class_map": None, "threshold":0.0, "dense_layers": [64,32,16,8], "drop_out": 0.1, "iterations": 5, "learning_rate": 0.01, "batch_size": 64, "loss_function":"categorical_crossentropy", "tensorflow_optimizer":"adam", "batch_normalization": True, "final_activation":"softmax, })
>>> client.pipeline.set_classifier("TensorFlow Lite for Microcontrollers")
>>> client.pipeline.set_tvo({'validation_seed':None})
>>> results, stats = client.pipeline.execute() >>> results.summarize()

xGBoost
Train an ensemble of boosted tree classifiers using the xGBoost training algorithm.
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
max_depth (int) – The max depth to allow a decision tree to reach
n_estimators (int) – The number of decision trees to build.
 Returns
a trained model

RBF with Neuron Allocation Limit
The Train and Prune algorithm takes as input feature vectors, corresponding class labels, and maximum desired number of neurons, and outputs a model.
The training vectors are partitioned into subsets (chunks) and presented to the PME classifier which places neurons and determines areas of influence (AIFs). After each subset is learned, the neurons that fired the most on the validation set are retained and the others are removed (pruned) from the model. After a defined number of train and prune cycles, the complete retained set of neurons is then relearned, which results in larger neuron AIFs. Train/prune/relearn cycles continue to run on all of the remaining chunks, keeping the total number of neurons within the limit while giving preference to neurons that fire frequently.
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
chunk_size (int) – the number of training vectors in each chunk
inverse_relearn_frequency (int) – the number of chunks to train and prune between each relearn phase
max_neurons (int) – the maximum allowed number of neurons
aggressive_neuron_creation (bool) – flag for placing neurons even if they are within the influence field of another neuron of the same category (default is False)
 Returns
a model

Random Forest
Train an ensemble of decision tree classifiers using the random forest training algorithm. A random forest is a meta estimator that fits a number of decision tree classifiers on various subsamples of the dataset and uses averaging to improve the predictive accuracy and control overfitting. The subsample size is always the same as the original input sample size but the samples are drawn with replacement
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
max_depth (int) – The max depth to allow a decision tree to reach
n_estimators (int) – The number of decision trees to build.
 Returns
a set of models

RBF with Neuron Allocation Optimization
RBF with Neuron Allocation Optimization takes as input feature vectors, corresponding class labels, and desired number of iterations (or trials), and outputs a set of models. For each iteration the input vectors are randomly shuffled and presented to the PME classifier which either places the pattern as a neuron or does not. When a neuron is placed, an area of influence (AIF) is determined based on the neuron’s proximity to other neurons in the model and their respective classes.
 Parameters
input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
number_of_iterations (int) – the number of times to shuffle the training set;
turbo (boolean) – a flag that when True runs through the set of unplaced feature vectors repeatedly until no new neurons are placed (default is True)
number_of_neurons (int) – the maximum allowed number of neurons; when the model reaches this number, the algorithm will stop training
aggressive_neuron_creation (bool) – flag for placing neurons even if they are within the influence field of another neuron of the same category (default is False)
ranking_metric (str) – Method to score models by when evaluating best candidate. Options: [f1_score, sensitivity, accuracy]
 Returns
a set of models