metafor (version 1.4-0)

permutest.rma.uni: Permutation Tests for rma.uni Objects

Description

The function carries out permutation tests for objects of class "rma.uni".

Usage

## S3 method for class 'rma.uni':
permutest(x, exact=FALSE, iter=1000, progbar=TRUE, 
          retpermdist=FALSE, digits=x$digits, \dots)

Arguments

x

an object of class "rma.uni".

exact

logical indicating whether an exact permutation test should be carried out or not (default is FALSE). See Details.

iter

integer to specify the number of iterations for the permutation test when not doing an exact test (default is 1000 iterations).

progbar

logical indicating whether a progress bar should be shown (default is TRUE).

retpermdist

logical indicating whether the permutation distributions of the test statistics should also be returned (default is FALSE).

digits

an integer specifying the number of decimal places to which the printed results should be rounded (the default is to take the value from the object).

...

other arguments.

Value

• An object of class "permutest.rma.uni". The object is a list containing the following components:
• pval
  p-value(s) based on the permutation test.
• QMp
  p-value for the omnibus test of coefficients based on the permutation test.
• zval.perm
  values of the test statistics of the coefficients under the various permutations (only when retpermdist=TRUE).
• QM.perm
  values of the test statistic for the omnibus test of coefficients under the various permutations (only when retpermdist=TRUE).
• ...
  some additional elements/values are passed on.
• The results are formated and printed with the print.permutest.rma.uni function. One can also use coef.permutest.rma.uni to obtain the table with the model coefficients, corresponding standard errors, test statistics, p-values, and confidence interval bounds.

Details

For models without moderators, the permutation test is carried out by permuting the signs of the observed effect sizes or outcomes. The (two-sided) p-value of the permutation test is then equal to two times the proportion of times that the test statistic under the permuted data is as extreme or more extreme than under the actually observed data. See Follmann and Proschan (1999) for more details. For models with moderators, the permutation test is carried out by permuting the rows of the design matrix. The (two-sided) p-value for a particular model coefficient is then equal to two times the proportion of times that the test statistic for the coefficient under the permuted data is as extreme or more extreme than under the actually observed data. Similarly, for the omnibus test, the p-value is the proportion of times that the test statistic for the omnibus test is as extreme or more extreme than the actually observed one. See Higgins and Thompson (2004) for more details. If exact=TRUE, the function will try to carry out an exact permutation test. An exact permutation test requires fitting the model to each possible permutation once. However, the number of possible permutations increases rapidly with the number of outcomes/studies (i.e., $k$), especially with respect to the possible number of permutations of the design matrix. For example, for $k=5$, there are only 120 possible permutations of the design matrix (32 possible permutations of the signs). For $k=8$, there are already 40,320 (256). And for $k=10$, there are 3,628,800 (1024). Therefore, going through all possible permutations may become infeasible. Instead of using an exact permutation test, one can set exact=FALSE. In that case, the function approximates the exact permutation-based p-value(s) by going through a smaller number (as specified by the iter argument) of random permutations. Therefore, running the function twice on the same data will then yield (slightly) different p-values. Setting iter sufficiently large ensures that the results become stable. If exact=FALSE and iter is actually larger than the number of iterations required for an exact permutation test, then an exact test will be carried out. Also, the function will terminate with an error if the number of iterations is too large. Finally, it is important to note that the p-values obtained with permutation tests cannot reach conventional levels of statistical significance (i.e., $\leq .05$) when $k$ is very small. In particular, for models without moderators, the smallest possible (two-sided) p-value is .0625 when $k=5$ and .03125 when $k=6$. Therefore, $k$ must be at least equal to 6 to reject the null hypothesis at $\alpha=.05$. For models with moderators, the smallest possible (two-sided) p-value for a particular model coefficient is .08333 when $k=4$ and .01667 when $k=5$. Therefore, $k$ must be at least equal to 5 to reject the null hypothesis at $\alpha=.05$.

References

Follmann, D. A. & Proschan, M. A. (1999). Valid inference in random effects meta-analysis. Biometrics, 55, 732--737. Good, P. I. (2009). Permutation, parametric, and bootstrap tests of hypotheses (3rd ed.). New York: Springer. Higgins, J. P. T. & Thompson, S. G. (2004). Controlling the risk of spurious findings from meta-regression. Statistics in Medicine, 23, 1663--1682. Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1--48. http://www.jstatsoft.org/v36/i03/.

See Also

rma.uni, print.permutest.rma.uni

Examples

### load BCG vaccine data
data(dat.bcg)

### calculate log relative risks and corresponding sampling variances
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, 
              data=dat.bcg, append=TRUE)

### random-effects model
res <- rma(yi, vi, data=dat, method="REML")

### permutation test (approximate and exact)
permutest(res)
permutest(res, exact=TRUE)

### mixed-effects model with two moderators (absolute latitude and publication year)
res <- rma(yi, vi, mods=cbind(ablat, year), data=dat, method="REML")

### permutation test (approximate only; exact not feasible)
permres <- permutest(res, iter=10000, retpermdist=TRUE)
permres

### histogram of permutation distribution for absolute latitude
### dashed horizontal line: the observed value of the test statistic
### red curve:  standard normal density
### blue curve: kernel density estimate of the permutation distribution
### note that the tail area under the permutation distribution is larger 
### than under a standard normal density (hence, the larger p-value)
hist(permres$zval.perm[,2], breaks=120, freq=FALSE, xlim=c(-5,5), ylim=c(0,.4),
     main="Permutation Distribution", xlab="Value of Test Statistic")
abline(v=res$zval[2], lwd=2, lty="dashed")
abline(v=0, lwd=2)
curve(dnorm, from=-5, to=5, add=TRUE, lwd=2, col=rgb(1,0,0,alpha=.7))
lines(density(permres$zval.perm[,2]), lwd=2, col=rgb(0,0,1,alpha=.7))