mlr (version 2.10)

estimateRelativeOverfitting: Estimate relative overfitting.

Description

Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.

Usage

estimateRelativeOverfitting(rdesc, measures, task, learner)
# S3 method for ResampleDesc
estimateRelativeOverfitting(rdesc, measures, task,
  learner)

Arguments

rdesc

[ResampleDesc] Resampling strategy.

measures

[Measure | list of Measure] Performance measure(s) to evaluate. Default is the default measure for the task, see here getDefaultMeasure.

task

[Task] The task.

learner

[Learner | character(1)] The learner. If you pass a string the learner will be created via makeLearner.

Value

[data.frame]. Relative overfitting estimate(s), named by measure(s), for each resampling iteration.

Details

Currently only support for classification and regression tasks is implemented.

References

Bradley Efron and Robert Tibshirani; Improvements on Cross-Validation: The .632+ Bootstrap Method, Journal of the American Statistical Association, Vol. 92, No. 438. (Jun., 1997), pp. 548-560.

See Also

Other performance: ConfusionMatrix, calculateConfusionMatrix, calculateROCMeasures, makeCostMeasure, makeCustomResampledMeasure, makeMeasure, measures, performance

Examples

task = makeClassifTask(data = iris, target = "Species")
rdesc = makeResampleDesc("CV", iters = 2)
estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.knn"))
estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.lda"))