# 2.10. Training and validation sets¶

Important

First go through the Getting Started: Lennard-Jones Potential for Argon tutorial.

With ParAMS, you can have a validation set in addition to the training set.

The optimizer will minimize the loss function on the training set. During the parametrization, you can also monitor the loss function on the validation set (see figure above). This approach is used to prevent overfitting. As long as the loss function on the (appropriately chosen) validation set is decreasing similarly to the loss function on the training set, it means that there is likely no overfitting.

There are two ways to include a validation set in ParAMS:

1. An explicit validation set in the same format as the training set
2. (Scripting only:) Setting the validation option to some fraction (e.g. validation = 0.1), which will randomly split the original data_set into a training set and validation set.

## 2.10.1. Explicit validation set (validation_set.yaml)¶



### 2.10.1.3. Training and validation set results¶

On the Validation Set panel, you can see the predictions and loss contributions just as on the Training Set panel.

On the Graphs panel, you can plot results for the training set, validation set, or both.

In one of the graph drop-downs, choose Loss → loss
This plots the loss function for both the training and validation set as a function of evaluation number
In the Data From: drop-down, you can toggle whether to plot the training loss, validation loss, or both.

In this case, both the training set and validation set losses decrease, so there is no sign of overfitting.

You can also plot the root-mean-squared error (RMSE) or mean absolute error (MAE) as a function of evaluation for both the training and validation sets:

In one of the graph drop-downs, choose Stats → Forces
This plots the RMSE of the forces for the training and validation sets

Note

If you plot Stats → Energy then you do not get any validation set results, since there were no Energy entries in the validation set!

For scatter plots:

In one of the graph drop-downs, choose Forces.
This plots two curves with titles Training (134): forces and Validation (134): forces.

The (134) (in your case the number may be different) indicates for which evaluation number the parameters came from. Every set of parameters during the parametrization has a unique evaluation number.

The default plot is Best Training, which in this case corresponds to the parameters at evaluation 134.

In the Best Training drop-down, select Best Validation
This plots curves with titles Validation (140): forces and Training (140): forces.
Evaluation 140 corresponds to the parameters that gave the lowest loss function on the validation set.

Note

Only after the parametrization has finished (from convergence, max_evaluations, or timeout) will you get the validation set results for the best training set parameters and vice versa. While the parametrization is running you will only be able to plot results for the training set using the best training set parameters, and for the validation set using the best validation set parameters.

You can also plot the latest evaluation:

In the Best Validation drop-down, select Latest Training
This plots a curve with the title Training (149).
In the Latest Training drop-down, select Latest Validation
This plots a curve with the title Validation (145).

Here, the latest training set evaluation was done with a different set of parameters than the latest validation set parameters. This is because the validation set is only evaluated every 5 iterations (as specified by the eval_every keyword in the settings).

Tip

You double-click on a plot axis to access plot configuration settings.

The results for the validation set enter a directory called validation_set_results:

optimization/
├── settings_and_initial_data
├── summary.txt
├── training_set_results
├── validation_set_results


The layout is exactly the same as for training_set_results.

Important

The validation_set_results/best directory contains results for the parameters that had the lowest validation set loss. This is generally not the same parameters that give the lowest error on the training set (for which the results are in training_set_results/best).

The training_set_results/best and validation_set_results/best directories get updated as the parametrization progresses. If the parametrization finishes successfully, either from reaching convergence, max_evaluations, or timeout, then two extra “best” directories will be created:

 Directory Results for Parameters gave lowest loss for training_set_results/best training set training set training_set_results/validation_set_best_parameters training set validation set validation_set_results/best validation set validation set validation_set_results/training_set_best_parameters validation set training set

The following pairs of files should be identical:

• training_set_results/best/parameter_interface.yaml and validation_set_results/training_set_best_paramaters/parameter_interface.yaml
• training_set_results/best/evaluation.txt and validation_set_results/training_set_best_paramaters/evaluation.txt
• validation_set_results/best/parameter_interface.yaml and training_set_results/validation_set_best_paramaters/parameter_interface.yaml
• validation_set_results/best/evaluation.txt and training_set_results/validation_set_best_paramaters/evaluation.txt

## 2.10.2. Scripting: Random split of a dataset into training and validation sets¶

The below shows how to do a random split into training and validation sets using scripting. How to do it in the GUI was explained above.

Download LJ_Ar_example.zip, or make a copy of the directory $AMSHOME/scripting/scm/params/examples/LJ_Ar. These are the same files that were used in the first tutorial. Modify params.conf.py by adding the following to the bottom of the file: validation = 0.40  This will split the dataset such that roughly 40% of the dataset entries enter the validation set, and the remaining 60% enter the training set. You can then run the optimization as normal: "$AMSBIN/params" optimize


To find out which dataset entries entered the validation set or training set, you can find this information in the optimization/settings_and_initial_data/data_sets directory contains two files training_set.pkl.gz and validation_set.pkl.gz. You can load these files using params as Python library with the help of the Data Set class.