Computes the ROC curve from a familiarEnsemble.
'
extract_auc_data(
object,
data,
cl = NULL,
ensemble_method = waiver(),
detail_level = waiver(),
estimation_type = waiver(),
aggregate_results = waiver(),
confidence_level = waiver(),
bootstrap_ci_method = waiver(),
is_pre_processed = FALSE,
message_indent = 0L,
verbose = FALSE,
...
)A list with data.tables for single and ensemble model ROC curve data.
A familiarEnsemble object, which is an ensemble of one or more
familiarModel objects.
A dataObject object, data.table or data.frame that
constitutes the data that are assessed.
Cluster created using the parallel package. This cluster is then
used to speed up computation through parallellisation.
Method for ensembling predictions from models for the same sample. Available methods are:
median (default): Use the median of the predicted values as the ensemble
value for a sample.
mean: Use the mean of the predicted values as the ensemble value for a
sample.
(optional) Sets the level at which results are computed and aggregated.
ensemble: Results are computed at the ensemble level, i.e. over all
models in the ensemble. This means that, for example, bias-corrected
estimates of model performance are assessed by creating (at least) 20
bootstraps and computing the model performance of the ensemble model for
each bootstrap.
hybrid (default): Results are computed at the level of models in an
ensemble. This means that, for example, bias-corrected estimates of model
performance are directly computed using the models in the ensemble. If there
are at least 20 trained models in the ensemble, performance is computed for
each model, in contrast to ensemble where performance is computed for the
ensemble of models. If there are less than 20 trained models in the
ensemble, bootstraps are created so that at least 20 point estimates can be
made.
model: Results are computed at the model level. This means that, for
example, bias-corrected estimates of model performance are assessed by
creating (at least) 20 bootstraps and computing the performance of the model
for each bootstrap.
Note that each level of detail has a different interpretation for bootstrap
confidence intervals. For ensemble and model these are the confidence
intervals for the ensemble and an individual model, respectively. That is,
the confidence interval describes the range where an estimate produced by a
respective ensemble or model trained on a repeat of the experiment may be
found with the probability of the confidence level. For hybrid, it
represents the range where any single model trained on a repeat of the
experiment may be found with the probability of the confidence level. By
definition, confidence intervals obtained using hybrid are at least as
wide as those for ensemble. hybrid offers the correct interpretation if
the goal of the analysis is to assess the result of a single, unspecified,
model.
hybrid is generally computationally less expensive then ensemble, which
in turn is somewhat less expensive than model.
A non-default detail_level parameter can be specified for separate
evaluation steps by providing a parameter value in a named list with data
elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid").
This parameter can be set for the following data elements: auc_data,
decision_curve_analyis, model_performance, permutation_vimp,
ice_data, prediction_data and confusion_matrix.
(optional) Sets the type of estimation that should be possible. This has the following options:
point: Point estimates.
bias_correction or bc: Bias-corrected estimates. A bias-corrected
estimate is computed from (at least) 20 point estimates, and familiar may
bootstrap the data to create them.
bootstrap_confidence_interval or bci (default): Bias-corrected
estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The
number of point estimates required depends on the confidence_level
parameter, and familiar may bootstrap the data to create them.
As with detail_level, a non-default estimation_type parameter can be
specified for separate evaluation steps by providing a parameter value in a
named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following
data elements: auc_data, decision_curve_analyis, model_performance,
permutation_vimp, ice_data, and prediction_data.
(optional) Flag that signifies whether results
should be aggregated during evaluation. If estimation_type is
bias_correction or bc, aggregation leads to a single bias-corrected
estimate. If estimation_type is bootstrap_confidence_interval or bci,
aggregation leads to a single bias-corrected estimate with lower and upper
boundaries of the confidence interval. This has no effect if
estimation_type is point.
The default value is equal to TRUE except when assessing metrics to assess
model performance, as the default violin plot requires underlying data.
As with detail_level and estimation_type, a non-default
aggregate_results parameter can be specified for separate evaluation steps
by providing a parameter value in a named list with data elements, e.g.
list("auc_data"=TRUE, , "model_performance"=FALSE). This parameter exists
for the same elements as estimation_type.
(optional) Numeric value for the level at which
confidence intervals are determined. In the case bootstraps are used to
determine the confidence intervals bootstrap estimation, familiar uses the
rule of thumb \(n = 20 / ci.level\) to determine the number of required
bootstraps.
The default value is 0.95.
(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:
percentile (default): Confidence intervals obtained using the percentile
method.
bc: Bias-corrected confidence intervals.
Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.
Flag that indicates whether the data was already
pre-processed externally, e.g. normalised and clustered. Only used if the
data argument is a data.table or data.frame.
Number of indentation steps for messages shown during computation and extraction of various data elements.
Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.
Unused arguments.
This function also computes credibility intervals for the ROC curve
for the ensemble model, at the level of confidence_level. In the case of
multinomial outcomes, an AUC curve is computed per class in a
one-against-all fashion.
To allow plotting of multiple AUC curves in the same plot and the use of ensemble models, the AUC curve is evaluated at 0.01 (1-specificity) intervals.