vimp (version 2.1.0)

vimp_accuracy: Nonparametric Variable Importance Estimates: Classification accuracy

Description

Compute estimates of and confidence intervals for nonparametric difference in classification accuracy-based variable importance. This is a wrapper function for cv_vim, with type = "accuracy".

Usage

vimp_accuracy(
  Y,
  X,
  f1 = NULL,
  f2 = NULL,
  indx = 1,
  V = 10,
  weights = rep(1, length(Y)),
  run_regression = TRUE,
  SL.library = c("SL.glmnet", "SL.xgboost", "SL.mean"),
  alpha = 0.05,
  delta = 0,
  na.rm = FALSE,
  folds = NULL,
  stratified = TRUE,
  scale = "identity",
  ...
)

Arguments

Y

the outcome.

X

the covariates.

f1

the predicted values on validation data from a flexible estimation technique regressing Y on X in the training data; a list of length V, where each object is a set of predictions on the validation data.

f2

the predicted values on validation data from a flexible estimation technique regressing the fitted values in f1 on X withholding the columns in indx; a list of length V, where each object is a set of predictions on the validation data.

indx

the indices of the covariate(s) to calculate variable importance for; defaults to 1.

V

the number of folds for cross-validation, defaults to 10.

weights

weights for the computed influence curve (e.g., inverse probability weights for coarsened-at-random settings)

run_regression

if outcome Y and covariates X are passed to cv_vim, and run_regression is TRUE, then Super Learner will be used; otherwise, variable importance will be computed using the inputted fitted values.

SL.library

a character vector of learners to pass to SuperLearner, if f1 and f2 are Y and X, respectively. Defaults to SL.glmnet, SL.xgboost, and SL.mean.

alpha

the level to compute the confidence interval at. Defaults to 0.05, corresponding to a 95% confidence interval.

delta

the value of the \(\delta\)-null (i.e., testing if importance < \(\delta\)); defaults to 0.

na.rm

should we remove NA's in the outcome and fitted values in computation? (defaults to FALSE)

folds

the folds to use, if f1 and f2 are supplied.

stratified

if run_regression = TRUE, then should the generated folds be stratified based on the outcome (helps to ensure class balance across cross-validation folds)

scale

scale should CIs be computed on original ("identity") or logit ("logit") scale? (defaults to "identity")

...

other arguments to the estimation tool, see "See also".

Value

An object of classes vim and vim_accuracy. See Details for more information.

Details

In the interest of transparency, we return most of the calculations within the vim object. This results in a list containing:

• call - the call to vim

• s - the column(s) to calculate variable importance for

• SL.library - the library of learners passed to SuperLearner

• full_fit - the fitted values of the chosen method fit to the full data

• red_fit - the fitted values of the chosen method fit to the reduced data

• est - the estimated variable importance

• naive - the naive estimator of variable importance

• update - the influence curve-based update

• se - the standard error for the estimated variable importance

• ci - the \((1-\alpha) \times 100\)% confidence interval for the variable importance estimate

• full_mod - the object returned by the estimation procedure for the full data regression (if applicable)

• red_mod - the object returned by the estimation procedure for the reduced data regression (if applicable)

• alpha - the level, for confidence interval calculation

• y - the outcome

See Also

SuperLearner for specific usage of the SuperLearner function and package.

Examples

# NOT RUN {
library(SuperLearner)
library(ranger)
## generate the data
## generate X
p <- 2
n <- 100
x <- data.frame(replicate(p, stats::runif(n, -1, 1)))

## apply the function to the x's
f <- function(x) 0.5 + 0.3*x[1] + 0.2*x[2]
smooth <- apply(x, 1, function(z) f(z))

## generate Y ~ Normal (smooth, 1)
y <- matrix(rbinom(n, size = 1, prob = smooth))

## set up a library for SuperLearner
learners <- "SL.ranger"

## estimate (with a small number of folds, for illustration only)
est <- vimp_accuracy(y, x, indx = 2,
           alpha = 0.05, run_regression = TRUE,
           SL.library = learners, V = 2, cvControl = list(V = 2))

# }