precision: Measures of Precision and Shrinkage for Ridge Regression

Description

The goal of precision is to allow you to study the relationship between shrinkage of ridge regression coefficients and their precision directly by calculating measures of each.

Three measures of (inverse) precision based on the “size” of the covariance matrix of the parameters are calculated. Let \(V_k \equiv \text{Var}(\mathbf{\beta}_k)\) be the covariance matrix for a given ridge constant, and let \(\lambda_i , i= 1, \dots p\) be its eigenvalues. Then the variance (= 1/precision) measures are:

"det": \(\log | V_k | = \log \prod \lambda\) (with det.fun = "log", the default) or \(|V_k|^{1/p} =(\prod \lambda)^{1/p}\) (with det.fun = "root") measures the linearized volume of the covariance ellipsoid and corresponds conceptually to Wilks' Lambda criterion
"trace": \( \text{trace}( V_k ) = \sum \lambda\) corresponds conceptually to Pillai's trace criterion
"max.eig": \( \lambda_1 = \max (\lambda)\) corresponds to Roy's largest root criterion.

Two measures of shrinkage are also calculated:

norm.beta: the root mean square of the coefficient vector \(\lVert\mathbf{\beta}_k \rVert\), normalized to a maximum of 1.0 if normalize == TRUE (the default).
norm.diff: the root mean square of the difference from the OLS estimate \(\lVert \mathbf{\beta}_{\text{OLS}} - \mathbf{\beta}_k \rVert\). This measure is inversely related to norm.beta

A plot method, plot.precision facilitates making graphs of these quantities.

Usage

precision(object, det.fun, normalize, ...)

Value

An object of class c("precision", "data.frame") with the following columns:

lambda: The ridge constant
df: The equivalent effective degrees of freedom
det: The det.fun function of the determinant of the covariance matrix
trace: The trace of the covariance matrix
max.eig: Maximum eigen value of the covariance matrix
norm.beta: The root mean square of the estimated coefficients, possibly normalized
norm.diff: The root mean square of the difference between the OLS solution (lambda = 0) and ridge solutions

Arguments

object: An object of class ridge or lm
det.fun: Function to be applied to the determinants of the covariance matrices, one of c("log","root").
normalize: If TRUE the length of the coefficient vector \(\mathbf{\beta}_k\) is normalized to a maximum of 1.0.
...: Other arguments (currently unused)

Author

Michael Friendly

Examples

Run this code


longley.y <- longley[, "Employed"]
longley.X <- data.matrix(longley[, c(2:6,1)])

lambda <- c(0, 0.005, 0.01, 0.02, 0.04, 0.08)
lridge <- ridge(longley.y, longley.X, lambda=lambda)

# same, using formula interface
lridge <- ridge(Employed ~ GNP + Unemployed + Armed.Forces + Population + Year + GNP.deflator, 
		data=longley, lambda=lambda)

clr <- c("black", rainbow(length(lambda)-1, start=.6, end=.1))
coef(lridge)

(pdat <- precision(lridge))
# plot log |Var(b)| vs. length(beta)
with(pdat, {
	plot(norm.beta, det, type="b", 
	cex.lab=1.25, pch=16, cex=1.5, col=clr, lwd=2,
	xlab='shrinkage: ||b|| / max(||b||)',
	ylab='variance: log |Var(b)|')
	text(norm.beta, det, lambda, cex=1.25, pos=c(rep(2,length(lambda)-1),4))
	text(min(norm.beta), max(det), "Variance vs. Shrinkage", cex=1.5, pos=4)
	})

# plot trace[Var(b)] vs. length(beta)
with(pdat, {
	plot(norm.beta, trace, type="b",
	cex.lab=1.25, pch=16, cex=1.5, col=clr, lwd=2,
	xlab='shrinkage: ||b|| / max(||b||)',
	ylab='variance: trace [Var(b)]')
	text(norm.beta, trace, lambda, cex=1.25, pos=c(2, rep(4,length(lambda)-1)))
#	text(min(norm.beta), max(det), "Variance vs. Shrinkage", cex=1.5, pos=4)
	})