6.2.4.2. Optimizer Stoppers¶
A Stopper
will cause an optimizer to stop and a new one to start (according
to the OptimizerSelector). Stoppers can be used to define convergence
criteria, or to stop optimizers that are performing poorly compared to the best
optimizer.
Stoppers
also follow the general pattern.
Tip
ExitConditions
are different from Stoppers
. To understand the difference between them, see the Difference Between Stoppers and ExitConditions.
By default, the current best optimizer is never stopped even if it satisfies the Stopper
conditions, but this can be allowed with the ApplyStoppersToBestOptimizer
option:
ApplyStoppersToBestOptimizer
 Type
Bool
 Default value
No
 Description
By default the stoppers are not applied to the best optimizer (the one who has seen the best value thus far). This is because many stoppers are based on comparisons to the best optimizer, and in most scenarios one would like to keep a wellperforming optimizer alive. For some stopper configurations this paradigm does not make sense and we would prefer to apply the stoppers equally to all optimizers.
Stoppers
are evaluated after every loss function call by default, but this interval can be adjusted with the CheckStopperInterval
key:
CheckStopperInterval
 Type
Integer
 Default value
1
 Description
Number of loss function evaluations between evaluations of the stopper conditions.
You can specify multiple stopper conditions, and combine them using logical operators together with the StopperBooleanCombination
option.
StopperBooleanCombination
 Type
String
 GUI name
Combine stoppers
 Description
If multiple Stoppers are used this is required to indicate how their evaluations relate to one another. Use an integer to refer to a stopper (defined by order in input file). Recognizes the symbols: ( ) &  E.g. (1 & 2)  3. Defaults to an OR combination of all selected stoppers.
For example, to stop optimizers who have used more than 10000 function calls, or who have not improved their best value in a long time:
Stopper
Type MaxFunctionCalls
MaxFunctionCalls 10000
End
Stopper
Type BestFunctionValueUnmoving
BestFunctionValueUnmoving
Tolerance 0.1
End
End
StopperBooleanCombination 1  2
Available types of stoppers:
 BestFunctionValueUnmoving
Stop optimizers that haven’t improved their best point in a long time.
 CurrentFunctionValueUnmoving
Stop optimizers which have not changed their currently explored loss function value a lot in a long time.
 MaxFunctionCalls
Stop optimizers when they have evaluated the loss function a certain number of times.
 MaxInteroptimizerDistance
Stop optimizers too close together in parameter space.
 MaxSequentialInvalidPoints
Stop optimizers that are stuck in areas of parameter space which return nonfinite loss function values.
 MinStepSize
Stop optimizers that are taking very small steps between iterations.
 OptimizerType
For use with other conditions. Stop optimizers of a specific type.
 TimeAnnealing
For use with other conditions. Keep an optimizer alive for longer if it’s relatively new.
 ValidationWorsening
Stop optimizers which are evaluating points where the validation set loss value is worse than the average of several previously seen ones.
 ValueAnnealing
For use with other conditions. Keep an optimizer alive if it looks likely to overtake the best optimizer.
Below we provide a technical, nontechnical and graphical description for each of the available Stoppers
so that you can understand their function.
6.2.4.2.1. BestFunctionValueUnmoving¶
This Stopper
is designed to stop optimizers which are ‘lost’ and unable to improve the best loss value they have found.
For example, CMAES will occasionally not show any degree of convergence over many evaluations when handling tough ReaxFF reparameterizations (see example below).
Intention 
Input Block 

“Stop optimizers which have not improved
by more than 10% in 900 function evaluations.”

Stopper
Type BestFunctionValueUnmoving
BestFunctionValueUnmoving
NumberOfFunctionCalls 900
Tolerance 0.1
End
End

Stopper
BestFunctionValueUnmoving
 Type
Block
 Description
Return True if an optimizer’s best value has not improved significantly in a given amount of time.
NumberOfFunctionCalls
 Type
Integer
 Description
Number of function evaluations between comparison points.
Tolerance
 Type
Float
 Default value
0.0
 Description
Function tolerance fraction between 0 and 1.
6.2.4.2.2. CurrentFunctionValueUnmoving¶
This Stopper
stops optimizers which are fluctuating near a minimum but are unable to converge with their own internal conditions.
This is a very common problem with ReaxFF reparameterizations; optimizers are unable to converge due to numerical instability near minima.
Intention 
Input Block 

“Stop optimizers which do not change the
function values they are exploring by
more than 10% in 400 function evaluations.”

Stopper
Type CurrentFunctionValueUnmoving
CurrentFunctionValueUnmoving
NumberOfFunctionCalls 400
Tolerance 0.1
End
End

Stopper
CurrentFunctionValueUnmoving
 Type
Block
 GUI name
Current function value unmoving:
 Description
Return True if an optimizer’s current function value has not improved significantly in a given amount of time.
NumberOfFunctionCalls
 Type
Integer
 Description
Number of function evaluations between comparison points.
Tolerance
 Type
Float
 Default value
0.0
 Description
Function tolerance fraction between 0 and 1.
6.2.4.2.3. MaxFunctionCalls¶
Simple Stopper
to limit the number of function evaluations an optimizer is allowed.
This is distinct from the similar ExitCondition
MaxTotalFunctionCalls.
Intention 
Input Block 

“Stop an optimizer after it has
used 500 function evaluations.”

Stopper
Type MaxFunctionCalls
MaxFunctionCalls 500
End

Stopper
MaxFunctionCalls
 Type
Integer
 Description
Returns True after an optimizer has evaluated at least n points. Use in an AND combination with other stoppers to keep optimizers alive for at least some period of time, or alone or in an OR combination to shut them down after some period of time.
6.2.4.2.4. MaxInteroptimizerDistance¶
Having two optimizer explore the same minimum is a waste of resources, particularly if you would like to identify multiple minima.
This Stopper
shuts down optimizers which are very close to one another in parameter space.
Intention 
Input Block 

“Stop an optimizer if it is within
5% of another one in parameter space.”

Stopper
Type MaxInteroptimizerDistance
MaxInteroptimizerDistance
MaxRelativeDistance 0.05
CompareAllOptimizers No
End
End

In this example, we plot two optimizer trajectories in the 2D parameter space they are exploring. They both start at the same point, diverge in their initial exploration and then begin to move towards the same minimum. Out of the two, the optimizer that is stopped is the one that has the higher best loss function value.
In order to make the MaxInteroptimizerDistance
easier to set up, the cutoff threshold is made relative to the longest distance in the space.
This depends on the dimensionality of the problem, in most cases the parameters are scaled between 0 and 1 (as in the example below),
and the longest distance in the space is \(\sqrt{n}\) where \(n\) is the number of optimized parameters.
CompareAllOptimizers
determines which optimizers are compared by the Stopper
.
If No
then the best optimizer (the one that has seen the lowest value) is compared to all other living optimizers.
If Yes
then all optimizer combinations are tested.
Stopper
MaxInteroptimizerDistance
 Type
Block
 Description
Return True if two optimizers are evaluating points too close to one another in parameter space.
CompareAllOptimizers
 Type
Bool
 Default value
No
 Description
If Yes the distance between all optimizer combinations are compared, otherwise worse optimizers are only compared to the best one.
MaxRelativeDistance
 Type
Float
 Description
Fraction (between 0 and 1) of the maximum distance in the space (from the point at all lower bounds to the point at all upper bounds) below which the optimizers are deemed too close and the victim will be stopped.
6.2.4.2.5. MaxSequentialInvalidPoints¶
Loss functions can return nonfinite results if jobs are unable to converge or constraints are violated. Some optimizers may struggle to deal with nonfinite values, or be unable to escape from that region of parameter space.
Intention 
Input Block 

“Stop optimizers which are stuck in a
nonfinite region of parameter space
for more than 50 function evaluations.”

Stopper
Type MaxSequentialInvalidPoints
MaxSequentialInvalidPoints 50
End

Stopper
MaxSequentialInvalidPoints
 Type
Integer
 Default value
1
 Description
Return True if the optimizer has failed to find a finite function value in the last n iterations. Usually indicates a lost optimizer with poor starting point.
6.2.4.2.6. MinStepSize¶
Optimizers which are averaging very small step sizes should also be stopped. This is sometimes needed, instead of an internal optimizer step size conditions which typically look at single step sizes.
Intention 
Input Block 

“Stop an optimizer if it is
averaging steps of less than 5%
over the last 30 function evaluations.”

Stopper
Type MinStepSize
MinStepSize
NumberOfFunctionCalls 30
Tolerance 0.05
End
End

Stopper
MinStepSize
 Type
Block
 Description
Return True if an optimizer’s step size between evaluations is too small.
NumberOfFunctionCalls
 Type
Integer
 Description
Number of function evaluations between comparison points.
Tolerance
 Type
Float
 Default value
0.0
 Description
Fraction (between 0 and 1) of the maximum distance in the space (from the point at all lower bounds to the point at all upper bounds)
6.2.4.2.7. OptimizerType¶
This is an advanced Stopper
intended for use in combination with others.
Will be triggered if an optimizer is of a specific type.
The intended use is to configure different stopping criteria for different optimizers if multiple types are used.
Intention 
Input Block 

“Stop CMA Optimizers.”

Stopper
Type OptimizerType
OptimizerType CMAES
End

Stopper
OptimizerType
 Type
Multiple Choice
 Options
[AdaptiveRateMonteCarlo, CMAES, Nevergrad, Scipy, RandomSampling, GridSampling]
 Description
Intended for use with other stoppers to allow for specific hunting conditions based on the type of optimizer.
6.2.4.2.8. TimeAnnealing¶
This is an advanced Stopper
intended for use in combination with others.
Optimizers are stopped randomly with some probability based on the time they have been alive.
By using this Stopper
with other conditions one can ensure optimizers are only stopped after they have been given some time to explore.
TimeAnnealing
is resolved as follows:
where:
\(n_\text{b}\) is the number of function evaluations used by the best optimizer;
\(n_\text{t}\) is the number of function evaluations used by the optimizer we is considering stopping;
\(u\) is a uniformly randomly drawn test statistic (\(u ~ \mathscr{U}[0, u_\text{crit})\));
\(u_\text{crit}\) is the critical ratio of function evaluations below which a tested optimizer will not be stopped.
If the test is less than this number the optimizer is stopped.
Intention 
Input Block 

“Keep an optimizer alive at least until
it has evaluated half as many times as
the best optimizer.”

Stopper
Type TimeAnnealing
TimeAnnealing
CriticalRatio 0.5
End
End

The image below shows the probability of the Stopper
being triggered as a function of the number of function evaluations used by the tested optimizer.
In this example, the best optimizer (to which the tested optimizer is compared) has used 1000 function evaluations.
Stopper
TimeAnnealing
 Type
Block
 Description
Keeps optimizers alive based on how long they have been alive. Randomly keeps optimizers alive based on how long (in function evaluations) they have been active. The newer an optimizer is, the more likely it will pass the test and be kept alive. Used to temper very strict termination conditions.
CriticalRatio
 Type
Float
 Default value
1.0
 Description
Critical ratio of function calls between stopper and victim above which the victim is guaranteed to survive. Values lower than one are looser and allow the victim to survive even if it has been in operation longer than the stopper. Values higher than one are stricter and may stop the victim even if it has iterated fewer times than the stopper.
6.2.4.2.9. ValidationWorsening¶
If a validation set is being used, test its loss function value and stop the optimizer if this value is getting worse. This is useful to guard against overfitting, a common problem during reparameterization.
Intention 
Input Block 

“Stop an optimizer if the validation set is
getting worse over the last 1000 evaluations.
Allow for fluctuations of 1%.”

Stopper
Type ValidationUnmoving
ValidationUnmoving
NumberOfFunctionCalls 1000
Tolerance 0.01
End
End

Stopper
ValidationWorsening
 Type
Block
 Description
Return True if the loss value of the validation set is increasing.
NumberOfFunctionCalls
 Type
Integer
 Default value
1
 Description
Number of function evaluations between comparison points.
Tolerance
 Type
Float
 Default value
0.0
 Description
The loss value (f) is modified by: f * (1 + tol). This can also be used to ignore the effects of mild fluctuations.
6.2.4.2.10. ValueAnnealing¶
This is an advanced Stopper
intended for use in combination with others.
Optimizers are stopped randomly with some probability based on the best loss value they have seen.
The intended use is to keep optimizers alive which otherwise meet the stopping conditions but are close to being the best optimizer,
and thus have the potential to overtake it.
ValueAnnealing
is resolved as follows:
where:
\(f_b\) is the lowest function value seen by the best optimizer;
\(f_t\) is the lowest loss function value seen by the optimizer the Stopper is considering stopping;
\(s\) is the critical stop chance defined as the probability of stopping the tested optimizer which is twice as large as the best optimizer in absolute value;
\(u\) is a uniformly randomly drawn test statistic such that \(u ~ \mathscr{U}[0, 1)\).
If the test is smaller than \(u\), the optimizer is stopped.
Intention 
Input Block 

“Keep an optimizer alive if it
has a reasonable chance of overtaking
the best optimizer.”

Stopper
Type ValueAnnealing
ValueAnnealing
CriticalStopChance 0.5
End
End

Stopper
ValueAnnealing
 Type
Block
 Description
Intended for use with other stoppers. This condition is unlikely to stop a victim which has a loss function value very near the best seen thus far, but the probability of stopping increases with the difference between them.
CriticalStopChance
 Type
Float
 Default value
0.5
 Description
Value is the probability of stopping a victim which is twice as large as the stopper in absolute value. The default is 50%.