6.2.8. Output and logging
*************************

For the generation of checkpoint files, see Generate checkpoints. The
results directory is a shared key, documented here.

Most of the output files (the contents of "training_set_results/best",
"training_set_results/running_loss.txt", etc., are updated every
"LoggingInterval%General" evaluations.

The training set is logged at evaluation numbers that are multiples of
"LoggingInterval%General", or when the loss function reaches a new
lowest value, or when the validation is evaluated (see below).

If there is a validation set, then

* The validation set is evaluated approximately every "EvaluateEvery"
  evaluations (configured in the validation set's "DataSet" block),

* The value for "EvaluateEvery" is adjusted to be a multiple of
  "LoggingInterval%General" (the validation set is not evaluated if it
  is not logged),

* The validation set is not necessarily evaluated/logged for
  evaluation numbers that are exact multiples of
  "LoggingInterval%General". It uses the evaluation number per
  optimizer (when there are multiple optimizers), and even then only
  an approximate multiple is used.

* Whenever the validation is logged, so is the training set.

"LoggingInterval"
   Type:
      Block

   Description:
      Number of function evaluations between every log to file.

   "Flush"
      Type:
         Integer

      Default value:
         10

      Description:
         The number of function evaluations between flushes of the
         output streams to disk.                      If Flush =
         General, then files will be flushed after every logged call.

   "General"
      Type:
         Integer

      Default value:
         10

      Description:
         The number of function evaluations between writes to stdout,
         running_loss.txt, running_stats.txt and all *latest/* and
         *best/* files.                      This is also the interval
         with which the validation set will be evaluated and logged.

   "History"
      Type:
         Integer

      Default value:
         500

      Description:
         The number of function evaluations between saves to the
         history directory with copies of latest/.

   "Parameters"
      Type:
         Integer

      Default value:
         500

      Description:
         The number of function evaluations between writes to
         running_active_parameters.txt.

"GlompoLogging"
   Type:
      Bool

   Default value:
      Yes

   GUI name:
      Print manager logging messages:

   Description:
      Include status and progress information from the optimization
      manager in the printstreams.

"SplitPrintstreams"
   Type:
      Bool

   Default value:
      Yes

   GUI name:
      Split optimizer printstreams:

   Description:
      Split print statements from each optimizer to separate files.

"PrintStatusInterval"
   Type:
      Float

   Default value:
      600.0

   Unit:
      s

   Description:
      Number of seconds between printing of a status summary.

"GlompoSummaryFiles"
   Type:
      Multiple Choice

   Default value:
      None

   Options:
      [None, 1, 2, 3, 4]

   GUI name:
      GloMPO summary files:

   Description:
      Indicates what GloMPO-style outputs you would like saved to
      disk. Higher values also save all lower level information.
      Available options:                   • None: Nothing is saved.
      • 1: YAML file with summary info about the optimization
      settings, performance and the result.                   • 2: PNG
      file showing the trajectories of the optimizers.
      • 3: HDF5 file containing iteration history for each optimizer.
      • 4: More detailed HDF5 log including the residual results for
      each optimizer, data set and iteration.

See also: More details on the structure and contents of the HDF5 file.

"PLAMSWorkingDirectory"
   Type:
      String

   Default value:
   Description:
      Path to PLAMS working directory to temporarily hold Job results
      files.
