ModelResultSet

Model objects from a pipeline come back from the server organized into ModelResultSets and Configurations. A ModelResultSet represents everything returned by the pipeline and a Configuration is a subset pertaining to a single train-validate-optimize (TVO) configuration or combination. It is very common for a ModelResultSet to have a configuration containing many models. See the tutorials for more information about models, their metrics, and recognition capabilities.

Copyright 2017-2024 SensiML Corporation

This file is part of SensiML™ Piccolo AI™.

SensiML Piccolo AI is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

SensiML Piccolo AI is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with SensiML Piccolo AI. If not, see <https://www.gnu.org/licenses/>.

class sensiml.datamanager.modelresults.ModelMetrics(configuration, sandbox, index, model_result)

Base class for a model metrics object.

confusion_matrix_stats

comprehensive metrics returned for the model

Type

list[ConfusionMatrix]

train_set

indices of the input data that the model was trained with

Type

list

validation_set

indices of the input data that the model was validated with

Type

list

test_set

indices of the input data that the model was tested with

Type

list

debug

structure containing debug information for some models

Type

dict

neurons

model neuron array

Type

list[dict]

parameters

model parameters

Type

dict

knowledgepack

knowledgepack associated with the model

Type

KnowledgePack

property knowledgepack: KnowledgePack

The model’s KnowledgePack object

recognize_signal(capturefile: Optional[str] = None, stop_step: Optional[int] = False, datafile: Optional[str] = None, segmenter: Optional[Union[Segmenter, bool]] = True, lock: bool = True, silent: bool = True, platform: str = 'emulator', compare_labels: bool = False, **kwargs) → DataFrame

Sends a DataFrame of raw signals to be run through the feature generation pipeline and recognized.

Parameters

  • capturefile (str) – The name of a file uploaded through the data capture lab

  • datafile (str) – The name of an uploading datafile

  • platform (str) – “emulator” or “cloud”. The “emulator” will run compiled c code giving device exact results, the cloud runs similarly to training providing more flexibility in returning early results by setting the stop step.

  • stop_step (int) – for debugging, if you want to stop the pipeline at a particular step, set stop_step to its index

  • compare_labels (bool) – If there are labels for the input dataframe, use them to create a confusion matrix

  • segmenter (bool or FunctionCall) – to suppress or override the segmentation algorithm in the original pipeline, set this to False or a function call of type ‘segmenter’ (defaults to True)

  • lock (bool, True) – If True, waits for the result to return before releasing the ipython cell.

Returns

a dataframe containing the results of recognition and a dictionary containing the execution summary and the confusion_matrix when labels are provided.

  • execution_summary: a summary of which steps ran in the execution engine

  • confusion_matrix: a confusion matrix, only if the input data has a ‘Label’ column

Return type

(DataFrame, dict)

recognize_vectors(vectors: DataFrame) → DataFrame

Sends a DataFrame of feature vectors to the model’s KnowledgePack for recognition.

Parameters

  • vectors (DataFrame) – where each row is a feature vector with column headings named the same as

  • pipeline (the features generated by the) –

  • included (and label columns may be) –

Returns

contains the results of recognition, including predicted class, neuron ID, and distance

Return type

(DataFrame)

summarize(metrics_set: str = 'validation')

Prints a formatted summary of the model metrics

Parameters

metrics_set (optional[str]) – options are ‘validation’ (the default), ‘test’, and ‘train’

class sensiml.datamanager.modelresults.ModelResultSet(project: Project, sandbox: Sandbox)

Base class for a model result set object.

get_knowledgepack_by_index(config_index: int, model_index: int)

Returns the KnowledgePack of the given configuration index and model index.

sort_models(metric: str = 'accuracy', metrics_set: str = 'validation')

Sorts the models within all configurations by a given metric or property.

Parameters

  • metric (optional[str]) – options are ‘accuracy’ (the default), ‘index’, and ‘number_of_neurons’

  • metrics_set (optional[str]) – options are ‘validation’ (the default), ‘test’, and ‘train’

summarize(metrics_set: str = 'validation', report_number: int = 5)

Prints a basic summary of each configuration and its models.

Parameters

metrics_set (optional[str]) – options are ‘validation’ (the default), ‘test’, and ‘train’

Copyright 2017-2024 SensiML Corporation

This file is part of SensiML™ Piccolo AI™.

SensiML Piccolo AI is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

SensiML Piccolo AI is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with SensiML Piccolo AI. If not, see <https://www.gnu.org/licenses/>.

class sensiml.datamanager.confusion_matrix.ConfusionMatrix(metrics_dict)

This object is a representation of a confusion matrix that contains properties for statistics that can be generated from a confusion matrix as well as a DataFrame representation for easy viewing

pp_by_class_name(class_name)

Returns the positive predictivity of a given class in the confusion matrix

Parameters

class_name (str) – name of the class to return positive predictivity of

Returns

if class name exists in matrix return PP, else NaN

Return type

pp_by_class

se_by_class_name(class_name)

Returns the sensitivity of a given class in the confusion matrix

Parameters

class_name (str) – name of the class to return sensitivity of

Returns

if class name exists in matrix return SE, else NaN

Return type

se_by_class