pvalues: Getting p-values for fitted models
Description
One of the most frequently asked questions about
lme4 is "how do I calculate p-values for estimated
parameters?" Previous versions of lme4 provided
the mcmcsamp function, which efficiently generated
a Markov chain Monte Carlo sample from the posterior
distribution of the parameters, assuming flat (scaled
likelihood) priors. Due to difficulty in constructing a
version of mcmcsamp that was reliable even in
cases where the estimated random effect variances were
near zero (e.g.
https://stat.ethz.ch/pipermail/r-sig-mixed-models/2009q4/003115.html),
mcmcsamp has been withdrawn (or more precisely,
not updated to work with lme4 versions >=1.0.0). Many users, including users of the aovlmer.fnc function from
the languageR package which relies on mcmcsamp, will be
deeply disappointed by this lacuna. Users who need p-values have a
variety of options. In the list below, the methods marked MC
provide explicit model comparisons; CI denotes confidence
intervals; and P denotes parameter-level or sequential tests of
all effects in a model. The starred (*) suggestions provide
finite-size corrections (important when the number of groups is <50);
those marked (+) support GLMMs as well as LMMs.
- likelihood ratio tests via
anova or drop1 (MC,+)
- profile confidence intervals via
profile.merMod and
confint.merMod (CI,+)
- parametric bootstrap confidence intervals and model comparisons via
bootMer (or PBmodcomp in the
pbkrtest package) (MC/CI,*,+)
- for random effects, simulation tests via the
RLRsim package
(MC,*)
- for fixed effects, F tests via Kenward-Roger
approximation using
KRmodcomp from the
pbkrtest package (MC,*)
car::Anova and
lmerTest::anova provide wrappers for
Kenward-Roger-corrected tests using pbkrtest:
lmerTest::anova also provides t tests via the
Satterthwaite approximation (P,*) afex::mixed is another wrapper for
pbkrtest and anova providing
"Type 3" tests of all effects (P,*,+)
arm::sim, or bootMer, can be used
to compute confidence intervals on predictions. For glmer models, the summary output provides p-values
based on asymptotic Wald tests (P); while this is standard practice
for generalized linear models, these tests make assumptions both about
the shape of the log-likelihood surface and about the accuracy of
a chi-squared approximation to differences in log-likelihoods. When all else fails, don't forget to keep p-values in
perspective:
http://www.phdcomics.com/comics/archive.php?comicid=905