mlr3measures (version 0.3.0)

mauc_aunu: Multiclass AUC Scores

Description

Multiclass AUC measures.

• AUNU: AUC of each class against the rest, using the uniform class distribution. Computes the AUC treating a c-dimensional classifier as c two-dimensional 1-vs-rest classifiers, where classes are assumed to have uniform distribution, in order to have a measure which is independent of class distribution change (Fawcett 2001).

• AUNP: AUC of each class against the rest, using the a priori class distribution. Computes the AUC treating a c-dimensional classifier as c two-dimensional 1-vs-rest classifiers, taking into account the prior probability of each class (Fawcett 2001).

• AU1U: AUC of each class against each other, using the uniform class distribution. Computes something like the AUC of c(c - 1) binary classifiers (all possible pairwise combinations). See Hand (2001) for details.

• AU1P: AUC of each class against each other, using the a priori class distribution. Computes something like AUC of c(c - 1) binary classifiers while considering the a priori distribution of the classes as suggested in Ferri (2009). Note we deviate from the definition in Ferri (2009) by a factor of c. The person implementing this function and writing this very documentation right now cautions against using this measure because it is an imperfect generalization of AU1U.

Usage

mauc_aunu(truth, prob, na_value = NaN, ...)
mauc_aunp(truth, prob, na_value = NaN, ...)
mauc_au1u(truth, prob, na_value = NaN, ...)
mauc_au1p(truth, prob, na_value = NaN, ...)

Arguments

truth

:: factor() True (observed) labels. Must have the same levels and length as response.

prob

:: matrix() Matrix of predicted probabilities, each column is a vector of probabilities for a specific class label. Columns must be named with levels of truth.

na_value

:: numeric(1) Value that should be returned if the measure is not defined for the input (as described in the note). Default is NaN.

...

:: any Additional arguments. Currently ignored.

Value

Performance value as numeric(1).

Meta Information

• Type: "classif"

• Range: \([0, 1]\)

• Minimize: FALSE

• Required prediction: prob

References

Fawcett, Tom (2001). “Using rule sets to maximize ROC performance.” In Proceedings 2001 IEEE international conference on data mining, 131--138. IEEE. Ferri, C<U+00E9>sar, Hern<U+00E1>ndez-Orallo, Jos<U+00E9>, Modroiu, R (2009). “An experimental comparison of performance measures for classification.” Pattern Recognition Letters, 30(1), 27--38. 10.1016/j.patrec.2008.08.010. Hand, J D, Till, J R (2001). “A simple generalisation of the area under the ROC curve for multiple class classification problems.” Machine learning, 45(2), 171--186.

See Also

Other Classification Measures: acc(), bacc(), ce(), logloss(), mbrier()

Examples

# NOT RUN {
set.seed(1)
lvls = c("a", "b", "c")
truth = factor(sample(lvls, 10, replace = TRUE), levels = lvls)
prob = matrix(runif(3 * 10), ncol = 3)
colnames(prob) = levels(truth)
mauc_aunu(truth, prob)
# }