performanceResample: Uncertainty About Performance of a Classifier (EXPERIMENTAL)

Description

Use resampling to quantify uncertainties about the performance of one or several binary classifiers evaluated via cross-validation.

Usage

performanceResample(
  object,
  data = NULL,
  name.response = NULL,
  type.resampling = "permutation",
  n.resampling = 1000,
  fold.repetition = 0,
  conf.level = 0.95,
  cpus = 1,
  seed = NULL,
  trace = TRUE,
  filename = NULL,
  ...
)

Value

An S3 object of class performance.

Arguments

object: a glm or range object, or a list of such object.
data: [data.frame] the training data.
name.response: [character] The name of the response variable (i.e. the one containing the categories).
type.resampling: [character] Should non-parametric bootstrap ("bootstrap") or permutation of the outcome ("permutation") be used.
n.resampling: [integer,>0] Number of bootstrap samples or permutations.
fold.repetition: [integer,>0] Nnumber of folds used in the cross-validation. Should be strictly positive.
conf.level: [numeric, 0-1] confidence level for the confidence intervals.
cpus: [integer, >0] the number of CPU to use. If strictly greater than 1, resampling is perform in parallel.
seed: [integer, >0] Random number generator (RNG) state used when starting resampling. If NULL no state is set.
trace: [logical] Should the execution of the function be traced.
filename: [character] Prefix for the files containing each result.
...: arguments passed to performance.

Details

WARNING: using bootstrap after cross-validation may not provide valid variance/CI/p-value estimates.