ModelResultSet

Model objects from a pipeline come back from the server organized into ModelResultSets and Configurations. A ModelResultSet represents everything returned by the pipeline and a Configuration is a subset pertaining to a single train-validate-optimize (TVO) configuration or combination. It is very common for a ModelResultSet to have a configuration containing many models. See the tutorials for more information about models, their metrics, and recognition capabilities.

class sensiml.datamanager.modelresults.ModelResultSet(project, sandbox)

Base class for a model result set object.

get_knowledgepack_by_index(config_index, model_index)

Returns the KnowledgePack of the given configuration index and model index.

sort_models(metric='accuracy', metrics_set='validation')

Sorts the models within all configurations by a given metric or property.

Parameters
  • metric (optional[str]) – options are ‘accuracy’ (the default), ‘index’, and ‘number_of_neurons’

  • metrics_set (optional[str]) – options are ‘validation’ (the default), ‘test’, and ‘train’

summarize(metrics_set='validation', report_number=5)

Prints a basic summary of each configuration and its models.

Parameters

metrics_set (optional[str]) – options are ‘validation’ (the default), ‘test’, and ‘train’

class sensiml.datamanager.modelresults.ModelMetrics(configuration, sandbox, index, model_result)

Base class for a model metrics object.

confusion_matrix_stats

comprehensive metrics returned for the model

Type

list[ConfusionMatrix]

train_set

indices of the input data that the model was trained with

Type

list

validation_set

indices of the input data that the model was validated with

Type

list

test_set

indices of the input data that the model was tested with

Type

list

debug

structure containing debug information for some models

Type

dict

neurons

model neuron array

Type

list[dict]

parameters

model parameters

Type

dict

knowledgepack

knowledgepack associated with the model

Type

KnowledgePack

get_false_positive_indexes(class_name, metric_set='validation')

Returns the indicides of false positives in the validation set.

Parameters

class_name (string) – name of class to get indices for

get_positive_predictive_indexes(class_name, metric_set='validation')

Returns the indicides of positive predictive classes in the validation set.

Parameters

class_name (string) – name of class to get indices for

get_true_positive_indexes(class_name, metric_set='validation')

Returns the indicides of true positive classification in the validation set.

Parameters

class_name (string) – name of class to get indices for

property knowledgepack

The model’s KnowledgePack object

recognize_signal(capturefile=None, stop_step=False, datafile=None, segmenter=True, lock=True, silent=True, platform='emulator', compare_labels=False, **kwargs)

Sends a DataFrame of raw signals to be run through the feature generation pipeline and recognized.

Parameters
  • capturefile (str) – The name of a captured file uploaded throught the data capture lab

  • datafile (str) – The name of an uploading datafile

  • platform (str) – “emulator” or “cloud”. The “emulator” will run compiled c code giving device exact results, the cloud runs similary to training providing more flexibility in returning early results by setting the stop step.

  • stop_step (int) – for debugging, if you want to stop the pipeline at a particular step, set stop_step to its index

  • compare_labels (bool) – If there are labels for the input dataframe, use them to create a confusion matrix

  • segmenter (bool or FunctionCall) – to suppress or override the segmentation algorithm in the original pipeline, set this to False or a function call of type ‘segmenter’ (defaults to True)

  • lock (bool, True) – If True, waits for the result to return before releasing the ipython cell.

Returns

a dataframe containing the results of recognition and a dictionary containing the execution summary and the confusion_matrix when labels are provided.

  • execution_summary: a summary of which steps ran in the execution engine

  • confusion_matrix: a confusion matrix, only if the input data has a ‘Label’ column

Return type

(DataFrame, dict)

recognize_vectors(vectors)

Sends a DataFrame of feature vectors to the model’s KnowledgePack for recognition.

Parameters
  • vectors (DataFrame) – where each row is a feature vector with column headings named the same as

  • features generated by the pipeline (the) –

  • label columns may be included (and) –

Returns

contains the results of recognition, including predicted class, neuron ID, and distance

Return type

(DataFrame)

summarize(metrics_set='validation')

Prints a formatted summary of the model metrics

Parameters

metrics_set (optional[str]) – options are ‘validation’ (the default), ‘test’, and ‘train’

class sensiml.datamanager.confusion_matrix.ConfusionMatrix(metrics_dict)

This object is a representation of a confusion matrix that contains properties for statistics that can be generated from a confusion matrix as well as a DataFrame representation for easy viewing

pp_by_class_name(class_name)

Returns the positive predictivity of a given class in the confusion matrix

Parameters

class_name (str) – name of the class to return positive predictivity of

Returns

if class name exists in matrix return PP, else NaN

Return type

pp_by_class

se_by_class_name(class_name)

Returns the sensitivity of a given class in the confusion matrix

Parameters

class_name (str) – name of the class to return sensitivity of

Returns

if class name exists in matrix return SE, else NaN

Return type

se_by_class