Given two numeric vectors of data, the mean squared error and R-squared are calculated. For two factors, the overall agreement rate and Kappa are determined.
defaultSummary(data, lev = NULL, model = NULL)postResample(pred, obs)
twoClassSummary(data, lev = NULL, model = NULL)
mnLogLoss(data, lev = NULL, model = NULL)
multiClassSummary(data, lev = NULL, model = NULL)
prSummary(data, lev = NULL, model = NULL)
a data frame with columns obs and
pred for the observed and predicted outcomes. For metrics
that rely on class probabilities, such as
twoClassSummary, columns should also include predicted
probabilities for each class. See the classProbs argument
to trainControl.
a character vector of factors levels for the
response. In regression cases, this would be NULL.
a character string for the model name (as taken
from the method argument of train.
A vector of numeric data (could be a factor)
A vector of numeric data (could be a factor)
A vector of performance estimates.
postResample is meant to be used with apply
across a matrix. For numeric data the code checks to see if the
standard deviation of either vector is zero. If so, the
correlation between those samples is assigned a value of zero.
NA values are ignored everywhere.
Note that many models have more predictors (or parameters) than
data points, so the typical mean squared error denominator (n -
p) does not apply. Root mean squared error is calculated using
sqrt(mean((pred - obs)^2. Also, \(R^2\) is calculated
wither using as the square of the correlation between the
observed and predicted outcomes when form = "corr". when
form = "traditional", $$ R^2 = 1-\frac{\sum (y_i -
\hat{y}_i)^2}{\sum (y_i - \bar{y}_i)^2} $$. Mean absolute error
is calculated using mean(abs(pred-obs)).
defaultSummary is the default function to compute
performance metrics in train. It is a wrapper
around postResample. The first argument is data,
which is data.frame with columns named obs and
pred for the observed and predicted outcome values
(either numeric data for regression or character values for
classification). The second argument is lev, a character
string that has the outcome factor levels or NULL for a
regression model. The third parameter is model, which can
be used if a summary metric is specific to a model function. If
other columns from the data are required to compute the summary
statistics, but should not be used in the model, the
recipe method for train can be used.
twoClassSummary computes sensitivity, specificity and
the area under the ROC curve. mnLogLoss computes the
minus log-likelihood of the multinomial distribution (without
the constant term): $$ -logLoss = \frac{-1}{n}\sum_{i=1}^n
\sum_{j=1}^C y_{ij} \log(p_{ij}) $$ where the y values are
binary indicators for the classes and p are the predicted
class probabilities.
prSummary (for precision and recall) computes values for
the default 0.50 probability cutoff as well as the area under
the precision-recall curve across all cutoffs and is labelled as
"AUC" in the output. If assumes that the first level of
the factor variables corresponds to a relevant result but the
lev argument can be used to change this.
multiClassSummary computes some overall measures of for
performance (e.g. overall accuracy and the Kappa statistic) and
several averages of statistics calculated from "one-versus-all"
configurations. For example, if there are three classes, three
sets of sensitivity values are determined and the average is
reported with the name ("Mean_Sensitivity"). The same is true
for a number of statistics generated by
confusionMatrix. With two classes, the basic
sensitivity is reported with the name "Sensitivity".
To use twoClassSummary and/or mnLogLoss, the
classProbs argument of trainControl should
be TRUE. multiClassSummary can be used without
class probabilities but some statistics (e.g. overall log loss
and the average of per-class area under the ROC curves) will not
be in the result set.
Other functions can be used via the summaryFunction
argument of trainControl. Custom functions must
have the same arguments asdefaultSummary.
The function getTrainPerf returns a one row data frame
with the resampling results for the chosen model. The statistics
will have the prefix "Train" (i.e. "TrainROC").
There is also a column called "method" that echoes the
argument of the call to trainControl of the same
name.
Kvalseth. Cautionary note about \(R^2\). American Statistician (1985) vol. 39 (4) pp. 279-285
# NOT RUN {
predicted <- matrix(rnorm(50), ncol = 5)
observed <- rnorm(10)
apply(predicted, 2, postResample, obs = observed)
classes <- c("class1", "class2")
set.seed(1)
dat <- data.frame(obs = factor(sample(classes, 50, replace = TRUE)),
pred = factor(sample(classes, 50, replace = TRUE)),
class1 = runif(50))
dat$class2 <- 1 - dat$class1
defaultSummary(dat, lev = classes)
twoClassSummary(dat, lev = classes)
prSummary(dat, lev = classes)
mnLogLoss(dat, lev = classes)
# }
Run the code above in your browser using DataLab