6.2.6. Parallelization options
******************************

The "ParallelLevels" block provides fine control over how ParAMS
parallelizes the optimization at various levels. The most important
option is "ParallelLevels%Optimizations" which allows you to run
multiple optimizers in parallel.

The total number of cores required to satisfy a "ParallelLevels"
configuration is the *product* of all the levels. For example, the
unrealistic setup below implicitly expects a 6 \times 5 \times 4
\times 3 \times 2 = 720 core system:

   ParallelLevels
       Optimizations 6
       ParameterVectors 5
       Jobs 4
       Processes 3
       Threads 2
   End

This configuration will run 6 optimizers in parallel. Each optimizer
will evaluate 5 parameter sets in parallel. For each evaluation, 4
jobs will be evaluated simultaneously. For each job, 3 processes will
be started. For each process, 2 threads will be used.

Important:

  Configure the parallelization options with care. In general, we
  recommend only setting "ParallelLevels%Optimizations" as appropriate
  defaults are automatically determined for the other levels. Many
  optimizers do not support parallel function evaluations, and tuning
  lower levels is very dependent on the size and properties of your
  JobCollection.

   ParallelLevels
      Optimizations integer
      ParameterVectors integer
      Jobs integer
      Processes integer
      Threads integer
   End

"ParallelLevels"
   Type:
      Block

   GUI name:
      Parallelization distribution:

   Description:
      Distribution of threads/processes between the parallelization
      levels.

   "Optimizations"
      Type:
         Integer

      Default value:
         1

      GUI name:
         Number of parallel optimizers

      Description:
         Number of independent optimizers to run in parallel.

   "ParameterVectors"
      Type:
         Integer

      Default value:
         0

      GUI name:
         Loss function evaluations (per optimizer)

      Description:
         Number of parameter vectors to try in parallel for each
         optimizer iteration. This level of parallelism can only be
         used with optimizers that support parallel optimization!
         Default (0) will set this value to the number of cores on the
         system divided by the number of optimizers run in parallel,
         i.e., each optimizer will be given an equal share of the
         resources.

   "Jobs"
      Type:
         Integer

      Default value:
         0

      GUI name:
         Jobs (per loss function evaluation)

      Description:
         Number of JobCollection jobs to run in parallel for each loss
         function evaluation.

   "Processes"
      Type:
         Integer

      Default value:
         1

      GUI name:
         Processes (per Job)

      Description:
         Number of processes (MPI ranks) to spawn for each
         JobCollection job. This effectively sets the NSCM environment
         variable for each job.                      A value of *-1*
         will disable explicit setting of related variables. We
         recommend a value of *1* in almost all cases. A value greater
         than 1 would only be useful if you parametrize DFTB with a
         serial optimizer and have very few jobs in the job
         collection.

   "Threads"
      Type:
         Integer

      Default value:
         1

      GUI name:
         Threads (per Process)

      Description:
         Number of threads to use for each of the processes. This
         effectively set the OMP_NUM_THREADS environment variable.
         Note that the DFTB engine does not use threads, so the value
         of this variable would not have any effect. We recommend
         always leaving it at the default value of 1. Please consult
         the manual of the engine you are parameterizing.
         A value of *-1* will disable explicit setting of related
         variables.
