6.4. Task: Sensitivity

Task Sensitivity runs an HSIC sensitivity analysis quantifying the effect of active parameters on the loss function.

To understand this task better, see the tutorial on Parameter sensitivity analysis.

6.4.1. Generating Samples

The first step of the sensitivity analysis is obtaining i.i.d uniformly-distributed random samples of the parameter space.

Samples can be automatically generated as part of the sensitivity calculation by using the RunSampling key:

RunSampling
Type

Bool

Default value

No

Description

Produce a set of samples of the loss function and active parameters. Samples from the parameter space are drawn from a uniform random distribution. Such a set of samples serves as the input to the sensitivity calculation.

Note

If you chose to generate samples, then the shared collection keys must also be defined so that ParAMS can construct the loss function to sample.

If you generating samples, you may select the number and manner in which you would like to sample:

NumberSamples
Type

Integer

Default value

1000

GUI name

Generate n samples:

Description

Number of samples to generate during the sampling procedure.

RandomSeed
Type

Integer

Description

Random seed to use during the sampling procedure (for reproducibility).

SaveResiduals
Type

Bool

Default value

No

Description

During the sampling, save the individual difference between reference and predicted values for every sample and training set item. Required for the Reweight calculation, and will be automatically activated if the reweight calculation is requested. Saving and analyzing the residuals can provide valuable insight into your training set, but can quickly occupy a large amount of disk space. Only save the residuals if you would like to run the reweight calculation or have a particular reason to do so.

Tip

You can reuse the residuals to calculate new loss values:

  • with different loss functions;

  • after changes to weights/sigmas; and

  • after removing training set items.

This allows you to tweak and tailor your training set and run a new sensitivity calculation without having to resample.

6.4.2. Loading Samples

Instead of generating samples, they can be loaded:

SamplesDirectory
Type

String

Default value

Description

Path to an ‘optimization’ directory containing the results of a previously run sampling. First looks for a ‘glompo_log.h5’ file. If not found, will look for ‘running_loss.txt’ and ‘running_active_parameters.txt’ in a sub-directory. The sub-directory used will depend on the DataSet Name. For the Reweight calculation only a ‘glompo_log.h5’ file (with residuals) may be used.

6.4.3. Kernels

The most important configurable options are the kernels applied to the parameter values and loss values.

We generally recommend:

  • applying the Gaussian kernel to the parameter values (in order to capture dis/similarity between parameter sets);

  • applying the conjunctive-Gaussian kernel to the loss values (in order to focus the weight of the distribution on good minima); and

  • using polynomial or linear kernels only if you have a specific type of relationship/dependency you would like to investigate.

ParametersKernel
   Alpha float
   Gamma float
   Polynomial
      Order integer
      Shift float
   End
   Sigma float
   Type [Gaussian | ConjunctiveGaussian | Threshold | Polynomial | Linear]
End
ParametersKernel
Type

Block

Description

Kernel applied to the parameters for which sensitivity is being measured.

Alpha
Type

Float

Description

Cut-off parameter for the Threshold kernel between zero and one. All loss values are scaled by taking the logarithm and then adjusted to a range between zero and one. This parameter is a value within this scaled space.

Gamma
Type

Float

Default value

0.1

Description

Bandwidth parameter for the conjunctive-Gaussian kernel.

Polynomial
Type

Block

Description

Settings for the Polynomial kernel.

Order
Type

Integer

Default value

1

Description

Maximum order of the polynomial.

Shift
Type

Float

Default value

0.0

Description

Free parameter (≥ 0) trading off higher-order versus lower-order effects.

Sigma
Type

Float

Default value

0.3

Description

Bandwidth parameter for the Gaussian kernel.

Type
Type

Multiple Choice

Default value

Gaussian

Options

[Gaussian, ConjunctiveGaussian, Threshold, Polynomial, Linear]

Description

Name of the kernel to applied to the parameters for which sensitivity is being measured.

LossValuesKernel
Type

Block

Description

Kernel applied to the parameters for which sensitivity is being measured.

Alpha
Type

Float

Description

Cut-off parameter for the Threshold kernel between zero and one. All loss values are scaled by taking the logarithm and then adjusted to a range between zero and one. This parameter is a value within this scaled space.

Gamma
Type

Float

Default value

0.3

Description

Bandwidth parameter for the conjunctive-Gaussian kernel.

Polynomial
Type

Block

Description

Settings for the Polynomial kernel.

Order
Type

Integer

Default value

1

Description

Maximum order of the polynomial.

Shift
Type

Float

Default value

0.0

Description

Free parameter (≥ 0) trading off higher-order versus lower-order effects.

Sigma
Type

Float

Description

Bandwidth parameter for the Gaussian kernel. If not specified or -1, calculates a reasonable default based on the number of parameters being tested.

Type
Type

Multiple Choice

Default value

ConjunctiveGaussian

Options

[Gaussian, ConjunctiveGaussian, Threshold, Polynomial, Linear]

Description

Name of the kernel to applied to the parameters for which sensitivity is being measured.

6.4.4. Other settings

The sensitivity analysis can be run on either the training set or the validation set:

SetToAnalyze
Type

Multiple Choice

Default value

TrainingSet

Options

[TrainingSet, ValidationSet]

GUI name

Analyze:

Description

Name of the data set to use for the sensitivity analysis.

Generally, all of the samples generated are not used simultaneously in the sensitivity calculation. This is because the more samples used, the slower the calculation. It is also difficult to know if one has enough samples to capture an accurate approximation of the true sensitivity.

Therefore, suppose one has generated 10000 samples. It is often better to run 10 repeats (bootstraps) of the calculation with 1000 points each than it is to run a single calculation with 10000 points. The first configuration will calculate faster, and provide a spread of the data to evaluate the robustness of the result.

To specify the number of times you would like the calculation repeated:

NumberBootstraps
Type

Integer

Default value

1

GUI name

Repeat calculation n times:

Description

Number of repeats of the calculation with different sub-samples. A small spread from a large number of bootstraps provides confidence on the estimation of the sensitivity.

To specify how many points to use from our sample set in each calculation:

NumberCalculationSamples
Type

Integer

GUI name

Number of samples per repeat:

Description

Number of samples from the full set available to use in the calculation. If not specified or -1, uses all available points. For the sensitivity calculation, this will be redrawn for every bootstrap.

To specify how the points are chosen from our sample set:

SampleWithReplacement
Type

Bool

Default value

Yes

Description

Sample from the available data with or without replacement. This only has an effect if the number of samples for the calculation is less than the total number available otherwise replace is Yes by necessity.

To remove parameter sets which produced non-finite loss values:

FilterInfiniteValues
Type

Bool

Default value

Yes

Description

If Yes, removes points from the calculation with non-finite loss values. Non-finite points can cause numerical issues in the sensitivity calculation.

Finally, we have included an extension to the sensitivity calculation:

RunReweightCalculation
Type

Bool

Default value

No

Description

Run a more expensive sensitivity calculation that will also return suggested weights for the training set which will produce more balanced sensitivities between all the parameters. Note: The Gaussian kernel is recommended for the loss values kernel in this case.

Warning

The reweight calculation is experimental.

6.4.5. Technical details

For technical details of the sensitivity calculation see the API documentation