azuremlsdk (version 1.10.0)

median_stopping_policy: Define a median stopping policy for early termination of HyperDrive runs

Description

Median stopping is an early termination policy based on running averages of primary metrics reported by the runs. This policy computes running averages across all training runs and terminates runs whose performance is worse than the median of the running averages. Specifically, a run will be canceled at interval N if its best primary metric reported up to interval N is worse than the median of the running averages for intervals 1:N across all runs.

Usage

median_stopping_policy(evaluation_interval = 1L, delay_evaluation = 0L)

Arguments

evaluation_interval

An integer of the frequency for applying policy.

delay_evaluation

An integer of the number of intervals for which to delay the first evaluation.

Value

The MedianStoppingPolicy object.

Details

The median stopping policy takes the following optional configuration parameters:

• evaluation_interval: Optional. The frequency for applying the policy. Each time the training script logs the primary metric counts as one interval.

• delay_evaluation: Optional. The number of intervals to delay the policy evaluation. Use this parameter to avoid premature termination of training runs. If specified, the policy applies every multiple of evaluation_interval that is greater than or equal to delay_evaluation.

This policy is inspired from the research publication Google Vizier: A Service for Black-Box Optimization.

If you are looking for a conservative policy that provides savings without terminating promising jobs, you can use a MedianStoppingPolicy with evaluation_interval = 1 and delay_evaluation = 5. These are conservative settings that can provide approximately 25%-35% savings with no loss on the primary metric (based on our evaluation data).

Examples

# NOT RUN {
# In this example, the early termination policy is applied at every
# interval starting at evaluation interval 5. A run will be terminated at
# interval 5 if its best primary metric is worse than the median of the
# running averages over intervals 1:5 across all training runs
# }
# NOT RUN {
early_termination_policy = median_stopping_policy(evaluation_interval = 1L,
                                                  delay_evaluation = 5L)
# }