5.3. Generate reference values
******************************

In the previous tutorials, reference data was either:

* pre-calculated and provided as part of the tutorial,

* or, we generated the training data by setting up a job in AMSinput
  and importing the results in ParAMS.

You can also use ParAMS to calculate the reference values as part of
the parametrization. An example is illustrated in this tutorial.

This can be useful if you, for example:

* want to re-evaluate (parts of) the training set with a different
  reference method,

* have added a reaction energy without a reference value, or

* only have a few reference jobs that are relatively quick to
  calculate.

[image]

Note:

  In most situations, it is better to first run the reference
  calculations and then import them into ParAMS (i.e., by following
  the Import training data (GUI) or Import Training Data with
  ResultsImporter tutorials). That way you can:

  * run the reference calculations on multiple nodes,

  * store the reference calculations and results in arbitrary places
    on disk,

  * inspect the reference results and correct any errors before the
    parametrization, and

  * use the results importer for PESScans which generate plots.

  Reference value generation is ParAMS is primarily intended for re-
  parametrization of existing models, or when using small datasets /
  fast reference methods.


5.3.1. Prerequisites
====================

* Go through the Getting Started: Lennard-Jones tutorial.

* Make a copy of the example directory
  "$AMSHOME/scripting/scm/params/examples/LJ_Ar_no_reference_data".


5.3.2. The input files
======================


5.3.2.1. Training set without reference values
----------------------------------------------

-[ GUI ]-

   Open the ParAMS GUI: **SCM → ParAMS**
   File → Open the **job_collection.yaml** file in the example directory
   Switch to the **Training Set** panel
   Note that the **Values** column is empty: There are no reference values!

   [image]

The reference values will be calculated when you start the
parametrization.

-[ training_set.yaml ]-

Unlike the training_set.yaml file in the previous tutorial, this
example's training_set.yaml does not contain any reference values:

   ---
   dtype: DataSet
   version: '2022.101'
   ---
   Expression: energy('Ar32_frame001')-energy('Ar32_frame002')
   Weight: 1.0
   Sigma: 0.054422772491975996
   Unit: eV, 27.211386245988
   ---
   Expression: energy('Ar32_frame003')-energy('Ar32_frame002')
   Weight: 1.0
   Sigma: 0.054422772491975996
   Unit: eV, 27.211386245988
   ---
   Expression: forces('Ar32_frame001')
   Weight: 1.0
   Sigma: 0.15426620242897765
   Unit: eV/angstrom, 51.422067476325886
   ---
   Expression: forces('Ar32_frame002')
   Weight: 1.0
   Sigma: 0.15426620242897765
   Unit: eV/angstrom, 51.422067476325886
   ---
   Expression: forces('Ar32_frame003')
   Weight: 1.0
   Sigma: 0.15426620242897765
   Unit: eV/angstrom, 51.422067476325886
   ...

This tutorial shows you how you can calculate the reference values (in
units of *Unit*) with "params".


5.3.2.2. Jobs and Engines
-------------------------

All **Jobs** that are mentioned in training set entries without a
reference value, will be calculated before the parametrization starts
with the help of a **Reference Engine**.

-[ GUI ]-

   Switch to the **Jobs** panel
   The **Reference Engine** column contains entries with "dftb;;kspace;;quality;GammaOnly;model;GFN1-xTB;".

   [image]

This rather cryptic ID is a **reference engine id**.

   Switch to the **Engines** panel

   [image]

   Double-click in the **Detail** column where it says "Engine dftb   kspace ......".

   [image]

You then see the **Engine settings** in the top box. If you are
familiar with this type of input, you can directly edit it in the text
box.

   Below, you see the **Engine ID**. Change the ID to something that is easier to read, for example "GFN1xTB_GammaPoint".

   [image]

   Click OK
   This updates the Engine ID in the table

   [image]

Modifications to the engine will affect all jobs with that particular
reference engine.

To change the engine that is used for a job:

   Switch back to the **Jobs** panel

   [image]

   The column with reference engines has been updated to contain the ID you chose.
   Double-click on the "Ar32_frame002" job in the **Detail** column (where it says **SinglePoint + gradients**)

   [image]

   Click the **AMSinput** button next to **Reference Engine**.
   This brings up an AMSinput window on the [image: DFTBPanel][image] panel (if you do not have a DFTB license, choose an engine for which you have a valid license).
   The engine settings that will be used for the job are shown. For example, **KSpace** is set to **GammaOnly**.
   You can change the DFTB settings. For example, set **KSpace** to **Basic** and **Occupation** to **Fermi**

   [image]

   Close the AMSinput window
   In the dialog "Pass to ParAMS?" click **Yes**
   A window informs that you that related reference values will be cleared. In this tutorial there are no reference values yet anyway. Click OK
   This creates a new reference engine "GFN1xTB;2526569033". The numbers are a hash of the settings.

   [image]

   The "Ar32_frame002" job has the new reference engine. You can rename it on the Engines panel if you prefer.

In this way, you can create an arbitrary number of engines, or choose
already-created engines.

   Switch to the **Jobs** panel
   Double-click on the "Ar32_frame002" job in the **Detail** column (where it says **SinglePoint + gradients**)

   [image]

   In the **Reference Engine** drop-down, choose the original reference engine ("GFN1xTB_GammaPoint").

   [image]

   Click OK
   Verify that all jobs have the "GFN1xTB_GammaPoint" engine

-[ job_collection.yaml ]-

Unlike the previous tutorial, here each entry in the
job_collection.yaml has a **ReferenceEngineID**. For example, the
first entry is

   ---
   Engines: job_collection_engines.yaml
   dtype: JobCollection
   version: '2022.101'
   ---
   ID: 'Ar32_frame001'
   ReferenceEngineID: dftb;;kspace;;quality;GammaOnly;model;GFN1-xTB;
   AMSInput: |
      properties
        gradients Yes
      End
      system
        Atoms
                   Ar       5.1883477539      -0.4887475488       7.9660568076 
                   Ar       5.7991822399       0.4024595652       2.5103286966 
                   Ar       6.1338265157       5.5335946219       7.0874208384 
                   Ar       4.6137188191       5.9644505949       3.0942810122 
                   Ar       8.4186778390       7.6292969115       8.0729664423 
                   Ar       8.3937816110       8.6402371806       2.6057806799 
                   Ar       7.5320205143       1.7666481606       7.7525889818 
                   Ar       8.5630139885       2.0472039529       2.6380554086 
                   Ar       2.6892353632       7.8435284207       7.7883054306 
                   Ar       2.4061636915       7.5716025415       2.4535180075 
                   Ar       2.2485171283       2.9764130946       7.8589298904 
                   Ar       3.0711058946       1.8500587164       2.5620921469 
                   Ar       7.6655637500      -0.4865893003       0.0018797080 
                   Ar       7.7550067215      -0.0222821825       4.8528637785 
                   Ar       7.7157262425       4.6625079517      -0.3861722152 
                   Ar       7.7434900996       5.2619590353       4.2602386226 
                   Ar       3.4302237084      -0.2708640738       0.6280466620 
                   Ar       2.8648051689       0.6106220610       6.1208342905 
                   Ar       3.2529823775       5.7151788324      -0.2024448179 
                   Ar       2.0046357208       4.9353027402       5.4968740217 
                   Ar       0.9326855213       8.0600564695      -0.3181225099 
                   Ar      -0.5654205469       8.5703446434       5.8930973456 
                   Ar      -0.9561293133       2.1098403312      -0.0052667919 
                   Ar      -0.8081417664       3.2747992855       5.5295389610 
                   Ar       5.5571960244       7.5645919074       0.1312355350 
                   Ar       4.4530832384       7.6170633330       5.4810860433 
                   Ar       5.1235367625       2.7983577675      -0.3161069611 
                   Ar       5.2048439076       2.9885672135       4.5193274119 
                   Ar      -0.2535891591       0.0134355189       8.3061692970 
                   Ar       0.5614183785      -0.1927751317       3.2355155467 
                   Ar      -0.0234943080       5.0313863031       8.0451075074 
                   Ar      -0.4760138873       6.2617510830       2.5759742219 
        End
        Lattice
              10.5200000000     0.0000000000     0.0000000000
               0.0000000000    10.5200000000     0.0000000000
               0.0000000000     0.0000000000    10.5200000000

The **ReferenceEngineID** refers to an engine in
"job_collection_engines.yaml", which is an Engine Collection. Each
entry has a unique **ID**, and an **AMSInput** block containing
calculation settings for the AMS engine.

   ---
   Jobs: job_collection.yaml
   dtype: EngineCollection
   version: '2022.101'
   ---
   ID: 'dftb;;kspace;;quality;GammaOnly;model;GFN1-xTB;'
   AMSInput: |
      Engine dftb
        kspace
          quality GammaOnly
        End
        model GFN1-xTB
      EndEngine
   ...

In this example, there is only one reference engine. It has the *ID*
"dftb;;kspace;;quality;GammaOnly;model;GFN1-xTB;". The *ID* could be
any string. It does not affect the results, but should describe the
reference engine. Each job in the job collection with this
*ReferenceEngineID* will be evaluated with this reference engine.

The *AMSInput* affects the calculation. In this example, it sets up a
GFN1-xTB engine with Γ-point sampling. The *AMSInput* will be added
verbatim to the input to the reference job.

Note:

  If you do not have a DFTB license, change the "Engine" block in
  "job_collection_engines.yaml" to

     Engine ForceField
         Type UFF
     EndEngine

  to instead use a UFF force field as the reference method.


5.3.3. Calculate the reference values
=====================================

-[ GUI ]-

Select the **Generate Reference** task:

   Click [image: OptimizationPanel][image] to open the input panel
   Click [image: OptimizationPanel][image] again to open the dropdown of available tasks
   Select **Generate Reference**

Save and run the task:

   **File → Save As**
   Save with the name "calc_ref_values.params"
   **File → Run**. This brings up AMSjobs

   [image]

Once complete the GUI will automatically load the reference values:

   Switch back to the ParAMS window
   On the **Training Set** panel, the reference values have been added

   [image]

Since the reference data has been automatically loaded, you may now
immediately continue to set up an optimization. You can switch the
task back to **Optimization**, change any settings you like (for
details see Getting Started: Lennard-Jones), and then save the job.
The reference values will be saved within the new job folder as
normal.

-[ Command-line ]-

To run the reference jobs and generate the reference data, change to
the example directory and run:

   "$AMSBIN/params"

This runs the reference calculations, adds the reference data to the
training set, and saves it in "results/training_set.ref.yaml"

To use the results in an optimization simply set one up as described
in Getting Started: Lennard-Jones and ensure that the "DataSet%Path"
key points to "results/training_set.ref.yaml" or a copy thereof.


5.3.4. Output files for reference calculations and data
=======================================================

In the results folder, you will find:

* the individual job results saved in "results/reference_jobs"

* the task input saved in "results/settings_and_initial_data"

* a file "training_set.ref.yaml" containing the reference values.


5.3.4.1. The reference_jobs folder
----------------------------------

For example, the file "reference_jobs/Ar32_frame001/Ar32_frame001.in"
contains the input to the "Ar32_frame001" job, which combines input
from the job collection and engine collection:

   properties
     gradients yes
   End

   system
     Atoms
                Ar       5.1883477539      -0.4887475488       7.9660568076 
                Ar       5.7991822399       0.4024595652       2.5103286966 
                Ar       6.1338265157       5.5335946219       7.0874208384 
                Ar       4.6137188191       5.9644505949       3.0942810122 
                Ar       8.4186778390       7.6292969115       8.0729664423 
                Ar       8.3937816110       8.6402371806       2.6057806799 
                Ar       7.5320205143       1.7666481606       7.7525889818 
                Ar       8.5630139885       2.0472039529       2.6380554086 
                Ar       2.6892353632       7.8435284207       7.7883054306 
                Ar       2.4061636915       7.5716025415       2.4535180075 
                Ar       2.2485171283       2.9764130946       7.8589298904 
                Ar       3.0711058946       1.8500587164       2.5620921469 
                Ar       7.6655637500      -0.4865893003       0.0018797080 
                Ar       7.7550067215      -0.0222821825       4.8528637785 
                Ar       7.7157262425       4.6625079517      -0.3861722152 
                Ar       7.7434900996       5.2619590353       4.2602386226 
                Ar       3.4302237084      -0.2708640738       0.6280466620 
                Ar       2.8648051689       0.6106220610       6.1208342905 
                Ar       3.2529823775       5.7151788324      -0.2024448179 
                Ar       2.0046357208       4.9353027402       5.4968740217 
                Ar       0.9326855213       8.0600564695      -0.3181225099 
                Ar      -0.5654205469       8.5703446434       5.8930973456 
                Ar      -0.9561293133       2.1098403312      -0.0052667919 
                Ar      -0.8081417664       3.2747992855       5.5295389610 
                Ar       5.5571960244       7.5645919074       0.1312355350 
                Ar       4.4530832384       7.6170633330       5.4810860433 
                Ar       5.1235367625       2.7983577675      -0.3161069611 
                Ar       5.2048439076       2.9885672135       4.5193274119 
                Ar      -0.2535891591       0.0134355189       8.3061692970 
                Ar       0.5614183785      -0.1927751317       3.2355155467 
                Ar      -0.0234943080       5.0313863031       8.0451075074 
                Ar      -0.4760138873       6.2617510830       2.5759742219 
     End
     Lattice
           10.5200000000     0.0000000000     0.0000000000
            0.0000000000    10.5200000000     0.0000000000
            0.0000000000     0.0000000000    10.5200000000
     End
   End

   task singlepoint

   Engine dftb
     kspace
       quality GammaOnly
     End
     model GFN1-xTB
   EndEngine


The normal AMS output can be found in the same folder: the logfile,
standard outputfile, and the binary ams.rkf and dftb.rkf files.


5.3.4.2. The training_set.ref.yaml file
---------------------------------------

The training_set.ref.yaml file contains the calculated reference
values. For example, it starts with

   ---
   dtype: DataSet
   version: '2022.101'
   ---
   Expression: energy('Ar32_frame001')-energy('Ar32_frame002')
   Weight: 1.0
   Sigma: 0.054422772491975996
   ReferenceValue: 0.20395942701637979
   Unit: eV, 27.211386245988
   ---
   Expression: energy('Ar32_frame003')-energy('Ar32_frame002')
   Weight: 1.0
   Sigma: 0.054422772491975996
   ReferenceValue: 0.22060005303998803
   Unit: eV, 27.211386245988
   ---
   Expression: forces('Ar32_frame001')
   Weight: 1.0
   Sigma: 0.15426620242897765
   ReferenceValue: |

These are the same reference values that were used in the Getting
Started: Lennard-Jones tutorial.


5.3.5. Generate reference values in Python
==========================================

See "run.py", where first the GenerateReference job is run, and then
the resulting reference values are used to run an optimization. See
also ParAMSJob and ParAMSResults.

   #!/usr/bin/env amspython
   from scm.plams import *
   from scm.params import *
   import os


   def main():
       init()

       inputfile = os.path.expandvars("$AMSHOME/scripting/scm/params/examples/LJ_Ar_no_reference_data/params.in")
       job = ParAMSJob.from_inputfile(inputfile, name="genref")
       job.run()

       # use all settings from the LJ_Ar example except for the path to the training set
       inputfile = os.path.expandvars("$AMSHOME/scripting/scm/params/examples/LJ_Ar/params.in")
       opt_job = ParAMSJob.from_inputfile(inputfile, name="optimization")
       opt_job.training_set = job.results.get_training_set_ref_path()
       opt_job.run()

       loss = opt_job.results.get_loss(source="latest")
       print(f"Final loss: {loss}")

       finish()


   if __name__ == "__main__":
       main()
