caret (version 4.42)

diff.resamples: Inferential Assessments About Model Performance

Description

Methods for making inferences about differences between models

Usage

## S3 method for class 'resamples':
diff(x, models = x$models, metric = x$metrics, test = t.test, ...)
## S3 method for class 'diff.resamples':
summary(object, digits = max(3, getOption("digits") - 3), ...)

Arguments

x

an object generated by resamples

models

a character string for which models to compare

metric

a character string for which metrics to compare

test

a function to compute differences. The output of this function should have scalar outputs called estimate and p.value

object

a object generated by diff.resamples

digits

the number of significant differences to display when printing

...

further arguments to pass to test

Value

• An object of class "diff.resamples" with elements:
• call
  the call
• difs
  a list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows
• statistics
  a list of results generated by test
• models
  a character string for which models were compared.
• metrics
  a character string of performance metrics that were used
• or...

  An object of class "summary.diff.resamples" with elements:

• call
  the call
• table
  a list of tables that show the differences and p-values

Details

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

See Also

resamples

Examples

#load(url("http://caret.r-forge.r-project.org/Classification_and_Regression_Training_files/exampleModels.RData"))

resamps <- resamples(list(CART = rpartFit,
                          CondInfTree = ctreeFit,
                          MARS = earthFit))

difs <- diff(resamps)

difs

summary(difs)