2.4.11 Training Algorithms

Training algorithms are used to select the optimal parameters of a model.

Hierarchical Clustering with Neuron Optimization

Hierarchical Clustering with Neuron Optimization takes as input feature vectors, corresponding class labels, and desired number of patterns, and outputs a model.

Each pattern in a model consists of a centroid, its class label, and its AIF. Each centroid is calculated as an average of objects in the cluster, each class label is the label of the majority class, and each AIF is the distance between the centroid and the farthest object in that cluster.

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
number_of_neurons (int) – the maximum number of output clusters (neurons) desired
linkage_method (str) – options are average, complete, ward, and single (default is average)
centroid_calculation (str) – options are robust, mean, and median (default is robust)
flip (int) – default is 1
cluster_method (str) – options are DLCH, DHC, and kmeans (default is DLHC)
aif_method (str) – options are min, max, robust, mean, median (default is max)
singleton_aif (int) – default is 0
min_number_of_dominant_vector (int) – It is used for pruning. It defines min. number of vector for dominant class in the cluster.
max_number_of_weak_vector (int) – It is used for pruning. It defines max. number of vector for weak class in the cluster.

Returns

one or more models

Load Model PME

Load Neuron Array takes an input of feature vectors, corresponding class labels, and a neuron array for classification. The neuron array is loaded, and classification is performed.

Note: This training algorithm does not perform optimizations on the provided neurons.

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
neuron_array (list) – A list of neurons to load into the hardware simulator.
class_map (dict) – class map for converting labels to neuron categories.

Returns

a set of models

Load Model TensorFlow Lite for Microcontrollers

Provides the ability to upload a TensorFlow Lite FlatBuffer library to use as the final classifier step in a pipeline.

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
model_parameters (int) – The FlatBuffer object of your TensorFlow micro model
class_map (dict) – class map for converting labels to output
estimator_type (str) – defines if this estimator performs regression or classification, defaults to classification
threshold (float) – if no values are greater than the threshold, classify as Unknown
train_history (dict) – training history for this model
model_json (dict) – expects the model JSON file from the TensorFlow APItf_model.to_json()

Example

MPLAB Machine Learning Development Suite provides the ability to train and bring your own NN architecture to use as the classifier for your pipeline. This example starts from the point where you have created features using the ML Model Builder Toolkit.

>>> x_train, x_test, x_validate, y_train, y_test, y_validate, class_map =             >>>     client.pipeline.features_to_tensor(fv_t, test=0.0, validate=.1)

TensorFlow Lite Micro only supports a subset of the full tensorflow functions. For a full list of available functions see the unhandled xref case. Use the Keras TensorFlow API to create the NN graph.

>>> from tensorflow.keras import layers
>>> import tensorflow as tf
>>> tf_model = tf.keras.Sequential()
>>> tf_model.add(layers.Dense(12, activation='relu',kernel_regularizer='l1', input_shape=(x_train.shape[1],)))
>>> tf_model.add(layers.Dropout(0.1))
>>> tf_model.add(layers.Dense(8, activation='relu', input_shape=(x_train.shape[1],)))
>>> tf_model.add(layers.Dropout(0.1))
>>> tf_model.add(layers.Dense(y_train.shape[1], activation='softmax'))
>>> tf_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
>>> tf_model.summary()
>>> train_history = {'loss':[], 'val_loss':[], 'accuracy':[], 'val_accuracy':[]}

Train the Tensorflow Model

>>> epochs=100
>>> batch_size=32
>>> data  = tf.data.Dataset.from_tensor_slices((x_train, y_train))
>>> shuffle_ds = data.shuffle(buffer_size=x_train.shape[0], reshuffle_each_iteration=True).batch(batch_size)
>>> history = tf_model.fit( shuffle_ds, epochs=epochs, batch_size=batch_size, validation_data=(x_validate, y_validate), verbose=0)
>>> for key in train_history:
>>>     train_history[key].extend(history.history[key])
>>> import mplabml.tensorflow.utils as sml_tf
>>> sml_tf.plot_training_results(tf_model, train_history, x_train, y_train, x_validate, y_validate)

Qunatize the Tensorflow Model

The `representative_dataset_generator()` function is necessary to provide statistical information about your dataset in order to quantize the model weights appropriatley.
The TFLiteConverter is used to convert a tensorflow model into a TensorFlow Lite model. The TensorFlow Lite model is stored as a unhandled xref case which, allows us to easily store and access it on embedded systems.
Quantizing the model allows TensorFlow Lite micro to take advantage of specialized instructions on cortex-M class processors using the unhandled xref case DSP library which, gives another huge boost in performance.

Quantizing the model can reduce size by up to 4x as 4 byte floats are converted to 1 byte ints in a number of places within the model.

>>> import numpy as np
>>> def representative_dataset_generator():
>>>     for value in x_validate:
>>>     # Each scalar value must be inside of a 2D array that is wrapped in a list
>>>         yield [np.array(value, dtype=np.float32, ndmin=2)]
>>>
>>> converter = tf.lite.TFLiteConverter.from_keras_model(tf_model)
>>> converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
>>> converter.representative_dataset = representative_dataset_generator
>>> tflite_model_quant = converter.convert()

Uploading Trained TF Lite model to MPLAB® Machine Learning Development Suite

>>> class_map_tmp = {k:v+1 for k,v in class_map.items()} #increment by 1 as 0 corresponds to unknown
>>> client.pipeline.set_training_algorithm("Load Model TensorFlow Lite for Microcontrollers",
>>>                                     params={"model_parameters": {
>>>                                             'tflite': sml_tf.convert_tf_lite(tflite_model_quant)},
>>>                                             "class_map": class_map_tmp,
>>>                                             "estimator_type": "classification",
>>>                                             "threshold": 0.0,
>>>                                             "train_history":train_history,
>>>                                             "model_json": tf_model.to_json()
>>>                                             })
>>> client.pipeline.set_validation_method("Recall", params={})
>>> client.pipeline.set_classifier("TensorFlow Lite for Microcontrollers", params={})
>>> client.pipeline.set_tvo()
>>> results, stats = client.pipeline.execute()
>>>
>>> results.summarize()

Neuron Optimization

Neuron Optimization performs an optimized grid search over KNN/RBF and the number of neurons for the parameters of Hierarchical Clustering with Neuron Optimization. Takes as input feature vectors, corresponding class labels and outputs a model.

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
neuron_range (list) – the range of max neurons spaces to search over specified as [Min, Max]
linkage_method (str) – options are average, complete, ward, and single (default is average)
centroid_calculation (str) – options are robust, mean, and median (default is robust)
flip (int) – default is 1
cluster_method (str) – options are DLCH, DHC, and kmeans (default is DLHC)
aif_method (str) – options are min, max, robust, mean, median (default is max)
singleton_aif (int) – default is 0
min_number_of_dominant_vector (int) – It is used for pruning. It defines min. number of vector for dominant class in the cluster.
max_number_of_weak_vector (int) – It is used for pruning. It defines max. number of vector for weak class in the cluster.

Returns

one or more models

“description”: “.”}

return None

Bonsai Tree Optimizer

Train a Bonsais Tree Classifier using backpropagation.

For detailed information see the unhandled xref case

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
epochs (str) – The number of training epochs to iterate over
batch_size (float) – The size of batches to use during training
learning_rate (float) – The learning rate for training optimization
project_dimensions (int) – The number of dimensions to project the input feature space into
sigma (float) – tunable hyperparameter
reg_W (float) – regularization for W matrix
reg_V (float) – regularization for V matrix
reg_Theta (float) – regularization for Theta matrix
reg_Z (float) – regularization for Z matrix
sparse_V (float) – sparcity factor for V matrix
sparse_Theta (float) – sparcity factor for Theta matrix
sparse_W (float) – sparcity factor for W matrix
sparse_Z (float) – sparcity factor fo Z matrix

Returns

model parameters for a bonsai tree classifier

Train Fully Connected Neural Network

Provides the ability to train a fully connected neural network model to use as the final classifier step in a pipeline.

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
class_map (dict) – optional, class map for converting labels to output
estimator_type (str) – defines if this estimator performs regression or classification, defaults to classification
threshold (float) – if no values are greater than the threshold, classify as Unknown
dense_layers (list) – The size of each dense layer
drop_out (float) – The amount of dropout to use after each dense layer
batch_normalization (bool) – Use batch normalization
final_activation (str) – the final activation to use
iteration (int) – Maximum optimization attempt
batch_size (int) – The batch size to use during training
metrics (str) – the metric to use for reporting results
learning_rate (float) – The learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.
batch_size – Refers to the number of training examples utilized in one iteration.
loss_function (str) – It is a function that determine how far the predicted values deviate from the actual values in the training data.
tensorflow_optimizer (str) – Optimization algorithms that is used to minimize loss function.

Example

MPLAB® Machine Learning Development Suite provides the ability to train NN architecture to use as the classifier for your pipeline. TensorFlow Lite Micro only supports a subset of the full TensorFlow functions. For a full list of available functions see the unhandled xref case. Use the Keras TensorFlow API to create the NN graph.

>>> client.project = 'Activity_Detection'
>>> client.pipeline = 'tf_p1'

>>> client.pipeline.stop_pipeline()

>>> sensors = ['GyroscopeX', 'GyroscopeY', 'GyroscopeZ', 'AccelerometerX', 'AccelerometerY', 'AccelerometerZ']

>>> client.pipeline.reset()

>>> client.pipeline.set_input_query("Q1")

>>> client.pipeline.add_transform("Windowing", params={"window_size":200,
                                "delta":200,
                                "train_delta":0})

>>> client.pipeline.add_feature_generator([
        {'name':'MFCC', 'params':{"columns":sensors,"sample_rate":100, "cepstra_count":1}}
    ])

>>> client.pipeline.add_transform("Min Max Scale")

>>> client.pipeline.set_validation_method("Recall", params={})

>>> client.pipeline.set_training_algorithm("Train Fully Connected Neural Network", params={
                        "estimator_type":"classification",
                        "class_map": None,
                        "threshold":0.0,
                        "dense_layers": [64,32,16,8],
                        "drop_out": 0.1,
                        "iterations": 5,
                        "learning_rate": 0.01,
                        "batch_size": 64,
                        "loss_function":"categorical_crossentropy",
                        "tensorflow_optimizer":"adam",
                        "batch_normalization": True,
                        "final_activation":"softmax,
    })

>>> client.pipeline.set_classifier("TensorFlow Lite for Microcontrollers")

>>> client.pipeline.set_tvo({'validation_seed':None})

>>> results, stats = client.pipeline.execute()
>>> results.summarize()

xGBoost

Train an ensemble of boosted tree classifiers using the xGBoost training algorithm.

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
max_depth (int) – The max depth to allow a decision tree to reach
n_estimators (int) – The number of decision trees to build.

Returns

a trained model

RBF with Neuron Allocation Limit

The Train and Prune algorithm takes as input feature vectors, corresponding class labels, and maximum desired number of neurons, and outputs a model.

The training vectors are partitioned into subsets (chunks) and presented to the PME classifier which places neurons and determines AIFs. After each subset is learned, the neurons that fired the most on the validation set are retained, and the others are removed (pruned) from the model. After a defined number of train and prune cycles, the complete retained set of neurons is then re-learned, which results in larger neuron AIFs. Train/prune/re-learn cycles continue to run on all of the remaining chunks, keeping the total number of neurons within the limit, while giving preference to neurons that fire frequently.

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
chunk_size (int) – the number of training vectors in each chunk
inverse_relearn_frequency (int) – the number of chunks to train and prune between each re-learn phase
max_neurons (int) – the maximum allowed number of neurons
aggressive_neuron_creation (bool) – flag for placing neurons even if they are within the influence field of another neuron of the same category (default is False)

Returns

a model

Random Forest

Train an ensemble of decision tree classifiers using the random forest training algorithm. A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control overfitting. The sub-sample size is always the same as the original input sample size but the samples are drawn with replacement.

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
max_depth (int) – The max depth to allow a decision tree to reach
n_estimators (int) – The number of decision trees to build.

Returns

a set of models

RBF with Neuron Allocation Optimization

RBF with Neuron Allocation Optimization takes as input feature vectors, corresponding class labels, and desired number of iterations (or trials), and outputs a set of models. For each iteration the input vectors are randomly shuffled and presented to the PME classifier which either places the pattern as a neuron or does not. When a neuron is placed, an area of influence (AIF) is determined based on the neuron’s proximity to other neurons in the model and their respective classes.

Parameters

input_data (DataFrame) – input feature vectors with a label column
label_column (str) – the name of the column in input_data containing labels
number_of_iterations (int) – the number of times to shuffle the training set;
turbo (boolean) – a flag that when True runs through the set of unplaced feature vectors repeatedly until no new neurons are placed (default is True)
number_of_neurons (int) – the maximum allowed number of neurons; when the model reaches this number, the algorithm will stop training
aggressive_neuron_creation (bool) – flag for placing neurons even if they are within the influence field of another neuron of the same category (default is False)
ranking_metric (str) – Method to score models by when evaluating best candidate. Options: [f1_score, sensitivity, accuracy]

Returns

a set of models