6.2.8. Output and logging¶

For the generation of checkpoint files, see Generate checkpoints. The results directory is a shared key, documented here

Most of the output files (the contents of training_set_results/best, training_set_results/running_loss.txt, etc., are updated every LoggingInterval%General evaluations.

The training set is logged at evaluation numbers that are multiples of LoggingInterval%General, or when the loss function reaches a new lowest value, or when the validation is evaluated (see below).

If there is a validation set, then

The validation set is evaluated approximately every EvaluateEvery evaluations (configured in the validation set’s DataSet block),
The value for EvaluateEvery is adjusted to be a multiple of LoggingInterval%General (the validation set is not evaluated if it is not logged),
The validation set is not necessarily evaluated/logged for evaluation numbers that are exact multiples of LoggingInterval%General. It uses the evaluation-number-per-optimizer (when there are multiple optimizers), and even then only an approximate multiple is used.
Whenever the validation is logged, so is the training set.

LoggingInterval

Type: Block
Description: Number of function evaluations between every log to file.

Flush

Type: Integer
Default value: 10
Description: The number of function evaluations between flushes of the output streams to disk. If Flush = General, then files will be flushed after every logged call.

General

Type: Integer
Default value: 10
Description: The number of function evaluations between writes to stdout, running_loss.txt, running_stats.txt and all latest/ and best/ files. This is also the interval with which the validation set will be evaluated and logged.

History

Type: Integer
Default value: 500
Description: The number of function evaluations between saves to the history directory with copies of latest/.

Parameters

Type: Integer
Default value: 500
Description: The number of function evaluations between writes to running_active_parameters.txt.

GlompoLogging

Type: Bool
Default value: Yes
GUI name: Print manager logging messages:
Description: Include status and progress information from the optimization manager in the printstreams.

SplitPrintstreams

Type: Bool
Default value: Yes
GUI name: Split optimizer printstreams:
Description: Split print statements from each optimizer to separate files.

PrintStatusInterval

Type: Float
Default value: 600.0
Unit: s
Description: Number of seconds between printing of a status summary.

GlompoSummaryFiles

Type: Multiple Choice
Default value: None
Options: [None, 1, 2, 3, 4]
GUI name: GloMPO summary files:
Description: Indicates what GloMPO-style outputs you would like saved to disk. Higher values also save all lower level information. Available options: • None: Nothing is saved. • 1: YAML file with summary info about the optimization settings, performance and the result. • 2: PNG file showing the trajectories of the optimizers. • 3: HDF5 file containing iteration history for each optimizer. • 4: More detailed HDF5 log including the residual results for each optimizer, data set and iteration.