metrics

To allow logging of model performance during each epoch of training we provide an interface to compute metrics in an online (streaming) fashion. This is not only more memory/time efficient than loading all generated files and calculating performance after an epoch, but it allows for performance to be reported at each batch.

See StatefulMetric for details on how to define a new streaming metric.

• StatefulMetric

• Handler

The following are metrics provided by Morgana. See F0Model for example usage.

• Print

• TensorHistory

• Mean

• RMSE

• F0Distortion

• LF0Distortion

StatefulMetric

class morgana.metrics.StatefulMetric

Bases: object

Abstract class for accumulating information and calculating a result using current stored data.

Three abstract methods must be implemented so that a metric can be calculated in an online fashion.

• reset_state()

• accumulate()

• result()

reset_state(self, *args)

Creates any stateful variables and sets their initial values.

accumulate(self, *args, **kwargs)

Accumulates a batch of values into the stateful variables.

result(self, *args)

Calculates the current result using the stateful variables.

result_as_json(self, *args)

If the result is a torch.Tensor then it must be converted to np.ndarray to be saved as JSON.

Handler

This metric is used to maintain multiple collections of metrics, it is created automatically by morgana.experiment_builder.ExperimentBuilder and added to your model instance as an attribute called metrics.

class morgana.metrics.Handler(**metrics)

Bases: morgana.metrics.StatefulMetric

Container for running a set of metrics.

Parameters

metrics (dict[str, StatefulMetric]) – Name of metrics and StatefulMetric instances that will be assigned to all metric collections.

collections

Multiple collections, each containing a map of metrics. Metrics can be overlapping between collections. It is possible to have multiple collections as different training modes (train/valid) may involve different metrics.

Type

dict[str, dict[str, StatefulMetric]]

add_metrics(self, collections=('all', ), **kwargs)

Updates the given collections with all names and metrics in kwargs.

The new metrics will also be added to self.metrics, even if no collections are specified.

Parameters

• collections (str or list[str]) – Name(s) of collections to update with the given metrics.

• kwargs (dict[str, StatefulMetric]) – Name of metrics and StatefulMetric instances to be added.

add_collection(self, collection, from_collections=())

Creates a new collection, and copies the metrics of other collection(s).

Parameters

• collection (str) – Name of new collection.

• from_collections – Name(s) of collections from which to copy existing metrics.

accumulate(self, collection, **kwargs)

Accumulates to all metrics in kwargs.

Parameters

• collection (str) – Metrics in this collection will be updated.

• kwargs (dict[str, tuple]) – Names of metrics, and inputs to each metric’s accumulate function, e.g. a list of torch.Tensor.

result(self, collection='all', *args)

Gets the result for all metrics in the given collection.

results_as_json_dict(self, collection='all', prefix='')

Gets the result (in a JSON friendly format) for all metrics in the given collection.

results_as_str_dict(self, collection='all', prefix='')

Gets the result (as a dictionary of strings) for all metrics in the given collection.

Print

class morgana.metrics.Print

Bases: morgana.metrics.StatefulMetric

Class for printing the last reported value.

value

Most recent accumulated input.

History

class morgana.metrics.History(max_len=None)

Bases: morgana.metrics.StatefulMetric

Class for storing the history of any object.

Parameters

max_len (int) – Maximum length of the history being stored.

history

Tensor of (up to max_len) previous inputs.

TensorHistory

class morgana.metrics.TensorHistory(feat_dim, max_len=None, device=None)

Bases: morgana.metrics.StatefulMetric

Class for storing the history of a tensor.

Parameters

• feat_dim (int) – Dimensionality of each feature vector. If 0, then no axis is used for the feature dimension.

• max_len (int) – Maximum length of the history being stored.

• device (str) – Device (cpu/cuda) to use. If None, this will be inferred from the tensor.

history

Tensor of (up to max_len) previous inputs.

str_summary(self, result)

Summarises history of tensors using Gaussian parameters and range.

Mean

class morgana.metrics.Mean

Bases: morgana.metrics.StatefulMetric

Class for computing the mean in an online fashion.

sum

Sum of inputs.

count

Number of valid frames for all inputs.

accumulate(self, tensor, seq_len=None)

tensor much have shape [batch_size, seq_len, feat_dim].

RMSE

class morgana.metrics.RMSE

Bases: morgana.metrics.Mean

Class for computing RMSE in an online fashion.

sum

Sum of squared difference of target and pred inputs.

count

Number of valid frames for all inputs.

F0Distortion

class morgana.metrics.F0Distortion

Bases: morgana.metrics.RMSE

Class for computing the F0 RMSE in Hz in an online fashion.

sum

Sum of squared difference of target and pred inputs.

count

Number of valid frames when both target and pred are voiced.

LF0Distortion

class morgana.metrics.LF0Distortion

Bases: morgana.metrics.F0Distortion

Class for computing the F0 RMSE in Hz in an online fashion.

sum

Sum of squared difference of target and pred inputs.

count

Number of valid frames when both target and pred are voiced.