auto_tuner: Function for Automatic Tuning

Description

The AutoTuner wraps a mlr3::Learner and augments it with an automatic tuning process for a given set of hyperparameters. The auto_tuner() function creates an AutoTuner object.

Usage

auto_tuner(
  tuner,
  learner,
  resampling,
  measure = NULL,
  term_evals = NULL,
  term_time = NULL,
  terminator = NULL,
  search_space = NULL,
  store_tuning_instance = TRUE,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  allow_hotstart = FALSE,
  keep_hotstart_stack = FALSE,
  evaluate_default = FALSE,
  callbacks = list(),
  method
)

Value

AutoTuner.

Arguments

tuner: (Tuner)
Optimization algorithm.
learner: (mlr3::Learner)
Learner to tune.
resampling: (mlr3::Resampling)
Resampling that is used to evaluate the performance of the hyperparameter configurations. Uninstantiated resamplings are instantiated during construction so that all configurations are evaluated on the same data splits. Already instantiated resamplings are kept unchanged. Specialized Tuner change the resampling e.g. to evaluate a hyperparameter configuration on different data splits. This field, however, always returns the resampling passed in construction.
measure: (mlr3::Measure)
Measure to optimize. If NULL, default measure is used.
term_evals: (integer(1))
Number of allowed evaluations. Ignored if terminator is passed.
term_time: (integer(1))
Maximum allowed time in seconds. Ignored if terminator is passed.
terminator: (Terminator)
Stop criterion of the tuning process.
search_space: (paradox::ParamSet)
Hyperparameter search space. If NULL (default), the search space is constructed from the TuneToken of the learner's parameter set (learner$param_set).
store_tuning_instance: (logical(1))
If TRUE (default), stores the internally created TuningInstanceSingleCrit with all intermediate results in slot $tuning_instance.
store_benchmark_result: (logical(1))
If TRUE (default), store resample result of evaluated hyperparameter configurations in archive as mlr3::BenchmarkResult.
store_models: (logical(1))
If TRUE, fitted models are stored in the benchmark result (archive$benchmark_result). If store_benchmark_result = FALSE, models are only stored temporarily and not accessible after the tuning. This combination is needed for measures that require a model.
check_values: (logical(1))
If TRUE, hyperparameter values are checked before evaluation and performance scores after. If FALSE (default), values are unchecked but computational overhead is reduced.
allow_hotstart: (logical(1))
Allow to hotstart learners with previously fitted models. See also mlr3::HotstartStack. The learner must support hotstarting. Sets store_models = TRUE.
keep_hotstart_stack: (logical(1))
If TRUE, mlr3::HotstartStack is kept in $objective$hotstart_stack after tuning.
evaluate_default: (logical(1))
If TRUE, learner is evaluated with hyperparameters set to their default values at the start of the optimization.
callbacks: (list of CallbackTuning)
List of callbacks.
method: (character(1))
Deprecated. Use tuner instead.

Default Measures

If no measure is passed, the default measure is used. The default measure depends on the task type.

Task	Default Measure	Package
`"classif"`	`"classif.ce"`	mlr3
`"regr"`	`"regr.mse"`	mlr3
`"surv"`	`"surv.cindex"`	mlr3proba
`"dens"`	`"dens.logloss"`	mlr3proba
`"classif_st"`	`"classif.ce"`	mlr3spatial
`"regr_st"`	`"regr.mse"`	mlr3spatial
`"clust"`	`"clust.dunn"`	mlr3cluster

Resources

There are several sections about hyperparameter optimization in the mlr3book.

Automate the tuning.
Estimate the model performance with nested resampling.

The gallery features a collection of case studies and demos about optimization.

Nested Resampling

Nested resampling is performed by passing an AutoTuner to mlr3::resample() or mlr3::benchmark(). To access the inner resampling results, set store_tuning_instance = TRUE and execute mlr3::resample() or mlr3::benchmark() with store_models = TRUE (see examples). The mlr3::Resampling passed to the AutoTuner is meant to be the inner resampling, operating on the training set of an arbitrary outer resampling. For this reason, the inner resampling should be not instantiated. If an instantiated resampling is passed, the AutoTuner fails when a row id of the inner resampling is not present in the training set of the outer resampling.

Details

The AutoTuner is a mlr3::Learner which wraps another mlr3::Learner and performs the following steps during $train():

The hyperparameters of the wrapped (inner) learner are trained on the training data via resampling. The tuning can be specified by providing a Tuner, a bbotk::Terminator, a search space as paradox::ParamSet, a mlr3::Resampling and a mlr3::Measure.
The best found hyperparameter configuration is set as hyperparameters for the wrapped (inner) learner stored in at$learner. Access the tuned hyperparameters via at$tuning_result.
A final model is fit on the complete training data using the now parametrized wrapped learner. The respective model is available via field at$learner$model.

During $predict() the AutoTuner just calls the predict method of the wrapped (inner) learner. A set timeout is disabled while fitting the final model.

Examples

Run this code

at = auto_tuner(
  tuner = tnr("random_search"),
  learner = lrn("classif.rpart", cp = to_tune(1e-04, 1e-1, logscale = TRUE)),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

at$train(tsk("pima"))

Run the code above in your browser using DataLab