Computes U-statistics to test for whether predictor X1 is more
concordant than predictor X2, extending rcorr.cens. For
method=1, estimates the fraction of pairs for which the
x1 difference is more impressive than the x2
difference. For method=2, estimates the fraction of pairs for
which x1 is concordant with S but x2 is not.
For binary responses the function improveProb provides several
assessments of whether one set of predicted probabilities is better
than another, using the methods describe in
Pencina et al (2007). This involves NRI and IDI to test for
whether predictions from model x1 are significantly different
from those obtained from predictions from model x2. This is a
distinct improvement over comparing ROC areas, sensitivity, or
specificity.
rcorrp.cens(x1, x2, S, outx=FALSE, method=1)improveProb(x1, x2, y)
# S3 method for improveProb
print(x, digits=3, conf.int=.95, ...)
a vector of statistics for rcorrp.cens, or a list with class
improveProb of statistics for improveProb:
number of cases
number of events
number of non-events
mean of pairwise differences in probabilities for those with events and a pairwise difference of \(\mbox{probabilities}>0\)
mean of pairwise differences in probabilities for those without events and a pairwise difference of \(\mbox{probabilities}>0\)
mean of pairwise differences in probabilities for those with events and a pairwise difference of \(\mbox{probabilities}>0\)
mean of pairwise differences in probabilities for those without events and a pairwise difference of \(\mbox{probabilities}>0\)
Net Reclassification Index = \((pup.ev-pdown.ev)-(pup.ne-pdown.ne)\)
standard error of NRI
Z score for NRI
Net Reclassification Index = \(pup.ev-pdown.ev\)
SE of NRI of events
Z score for NRI of events
Net Reclassification Index = \(pup.ne-pdown.ne\)
SE of NRI of non-events
Z score for NRI of non-events
improvement in sensitivity
improvement in specificity
Integrated Discrimination Index
SE of IDI
Z score of IDI
first predictor (a probability, for improveProb)
second predictor (a probability, for improveProb)
a possibly right-censored Surv object. If
S is a vector instead, it is converted to a
Surv object and it is assumed that no
observations are censored.
set to TRUE to exclude pairs tied on x1 or x2
from consideration
see above
a binary 0/1 outcome variable
the result from improveProb
number of significant digits for use in printing the result of
improveProb
level for confidence limits
unused
Frank Harrell
Department of Biostatistics, Vanderbilt University
fh@fharrell.com
Scott Williams
Division of Radiation Oncology
Peter MacCallum Cancer Centre, Melbourne, Australia
scott.williams@petermac.org
If x1,x2 represent predictions from models, these
functions assume either that you are using a separate sample from the
one used to build the model, or that the amount of overfitting in
x1 equals the amount of overfitting in x2. An example
of the latter is giving both models equal opportunity to be complex so
that both models have the same number of effective degrees of freedom,
whether a predictor was included in the model or was screened out by a
variable selection scheme.
Note that in the first part of their paper, Pencina et al. presented measures that required binning the predicted probabilities. Those measures were then replaced with better continuous measures that are implementedhere.
Pencina MJ, D'Agostino Sr RB, D'Agostino Jr RB, Vasan RS (2008): Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat in Med 27:157-172. DOI: 10.1002/sim.2929
Pencina MJ, D'Agostino Sr RB, D'Agostino Jr RB, Vasan RS: Rejoinder: Comments on Integrated discrimination and net reclassification improvements-Practical advice. Stat in Med 2007; DOI: 10.1002/sim.3106
Pencina MJ, D'Agostino RB, Steyerberg EW (2011): Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat in Med 30:11-21; DOI: 10.1002/sim.4085
rcorr.cens, somers2,
Surv, val.prob,
concordance
set.seed(1)
library(survival)
x1 <- rnorm(400)
x2 <- x1 + rnorm(400)
d.time <- rexp(400) + (x1 - min(x1))
cens <- runif(400,.5,2)
death <- d.time <= cens
d.time <- pmin(d.time, cens)
rcorrp.cens(x1, x2, Surv(d.time, death))
#rcorrp.cens(x1, x2, y) ## no censoring
set.seed(1)
x1 <- runif(1000)
x2 <- runif(1000)
y <- sample(0:1, 1000, TRUE)
rcorrp.cens(x1, x2, y)
improveProb(x1, x2, y)
Run the code above in your browser using DataLab