TAM (version 2.13-15)

tam.modelfit: Model Fit Statistics in TAM

Description

The function tam.modelfit computes several model fit statistics. It includes the Q3 statistic (Yen, 1984) and an adjusted variant of it (see Details). Effect sizes of model fit (MADaQ3, \(MADRESIDCOV\), \(SRMR\)) are also available.

The function IRT.modelfit is a wrapper to tam.modelfit, but allows convenient model comparisons by using the CDM::IRT.compareModels function.

The tam.modelfit function can also be used for fitted models outside the TAM package by applying tam.modelfit.IRT or tam.modelfit.args.

The function tam.Q3 computes the \(Q_3\) statistic based on weighted likelihood estimates (see tam.wle).

Usage

tam.modelfit(tamobj, progress=TRUE)
# S3 method for tam.modelfit
summary(object,...)
# S3 method for tam.mml
IRT.modelfit(object, …)
# S3 method for tam.mml.3pl
IRT.modelfit(object, …)
# S3 method for tamaan
IRT.modelfit(object, …)
# S3 method for IRT.modelfit.tam.mml
summary(object, …)
# S3 method for IRT.modelfit.tam.mml.3pl
summary(object, …)
# S3 method for IRT.modelfit.tamaan
summary(object, …)
tam.modelfit.IRT( object, progress=TRUE )
tam.modelfit.args( resp, probs, theta, post, progress=TRUE )
tam.Q3(tamobj, ... )
# S3 method for tam.Q3
summary(object,...)

Arguments

tamobj

Object of class tam

progress

An optional logical indicating whether progress should be displayed

object

Object of class tam.modelfit (for summary) or objects for which IRT.data, IRT.irfprob and IRT.posterior have been defined (for tam.modelfit.IRT).

resp

Dataset with item responses

probs

Array with item response functions evaluated at theta

theta

Matrix with used \(\bold{\theta}\) grid

post

Individual posterior distribution

…

Further arguments to be passed

Value

A list with following entries

stat.MADaQ3

Global fit statistic MADaQ3 and global model test with p value obtained by Holm adjustment

chi2.stat

Data frame with chi square tests of conditional independence for every item pair (Chen & Thissen, 1997)

fitstat

Model fit statistics \(100 \cdot MADRESIDCOV\), \(SRMR\) and \(SRMSR\)

modelfit.test

Test statistic of global fit based on multiple testing correction of \(\chi^2\) statistics

stat.itempair

Q3 and adjusted Q3 statistic for all item pairs

residuals

Residuals

Q3.matr

Matrix of \(Q_3\) statistics

aQ3.matr

Matrix of adjusted \(Q_3\) statistics

Q3_summary

Summary of \(Q_3\) statistics

N_itempair

Sample size for each item pair

Details

For each item \(i\) and each person \(n\), residuals \(e_{ni}=X_{ni}-E(X_{ni})\) are computed. The expected value \(E(X_{ni})\) is obtained by integrating the individual posterior distribution.

The Q3 statistic of item pairs \(i\) and \(j\) is defined as the correlation \(Q3_{ij}=Cor( e_{ni}, e_{nj} )\). The residuals in tam.modelfit are calculated by integrating values of the individual posterior distribution. Residuals in tam.Q3 are calculated by using weighted likelihood estimates (WLEs) from tam.wle.

It is known that under local independence, the expected value of \(Q_3\) is slightly smaller than zero. Therefore, an adjusted Q3 statistic (aQ3; \(aQ3_{ij}\)) is computed by subtracting the average of all Q3 statistics from Q3. To control for multiple testing, a p value adjustment by the method of Holm (p.holm) is employed (see Chen, de la Torre & Zhang, 2013).

An effect size of model fit (MADaQ3) is defined as the average of absolute values of \(aQ3\) statistics. An equivalent statistic based on the \(Q_3\) statistic is similar to the standardized generalized dimensionality discrepancy measure (SGDDM; Levy, Xu, Yel & Svetina, 2015).

The SRMSR (standardized root mean square root of squared residuals, Maydeu-Olivaras, 2013) is based on comparing residual correlations of item pairs $$ SRMSR=\sqrt{ \frac{1}{ J(J-1)/2 } \sum_{i < j} ( r_{ij} - \hat{r}_{ij} )^2 } $$ Additionally, the SRMR is computed as $$ SRMR=\frac{1}{ J(J-1)/2 } \sum_{i < j} | r_{ij} - \hat{r}_{ij} | $$

The \(MADRESIDCOV\) statistic (McDonald & Mok, 1995) is based on comparing residual covariances of item pairs $$ MADRESIDCOV=\frac{1}{ J(J-1)/2 } \sum_{i < j} | c_{ij} - \hat{c}_{ij} | $$ This statistic is just multiplied by 100 in the output of this function.

References

Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123-140.

Chen, W., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.

Levy, R., Xu, Y., Yel, N., & Svetina, D. (2015). A standardized generalized dimensionality discrepancy measure and a standardized model-based covariance for dimensionality assessment for multidimensional models. Journal of Educational Measurement, 52(2), 144--158.

Maydeu-Olivares, A. (2013). Goodness-of-fit assessment of item response theory models (with discussion). Measurement: Interdisciplinary Research and Perspectives, 11, 71-137.

McDonald, R. P., & Mok, M. M.-C. (1995). Goodness of fit in item response models. Multivariate Behavioral Research, 30, 23-40.

Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.

Examples

# NOT RUN {
#############################################################################
# EXAMPLE 1: data.cqc01
#############################################################################

data(data.cqc01)
dat <- data.cqc01

#*****************************************************
#*** Model 1: Rasch model
mod1 <- TAM::tam.mml( dat )
# assess model fit
res1 <- TAM::tam.modelfit( tamobj=mod1 )
summary(res1)
# display item pairs with five largest adjusted Q3 statistics
res1$stat.itempair[1:5,c("item1","item2","aQ3","p","p.holm")]

# }
# NOT RUN {
# IRT.modelfit
fmod1 <- IRT.modelfit(mod1)
summary(fmod1)

#*****************************************************
#*** Model 2: 2PL model
mod2 <- TAM::tam.mml.2pl( dat )
# IRT.modelfit
fmod2 <- IRT.modelfit(mod2)
summary(fmod2)

# model comparison
IRT.compareModels(fmod1, fmod2 )

#############################################################################
# SIMULATED EXAMPLE 2: Rasch model
#############################################################################

set.seed(8766)
N <- 1000    # number of persons
I <- 20      # number of items
# simulate responses
library(sirt)
dat <- sirt::sim.raschtype( rnorm(N), b=seq(-1.5,1.5,len=I) )
#*** estimation
mod1 <- TAM::tam.mml( dat )
summary(dat)
#*** model fit
res1 <- TAM::tam.modelfit( tamobj=mod1)
summary(res1)

#############################################################################
# EXAMPLE 3: Model fit data.gpcm | Partial credit model
#############################################################################

data(data.gpcm)
dat <- data.gpcm

# estimate partial credit model
mod1 <- TAM::tam.mml( dat)
summary(mod1)

# assess model fit
tmod1 <- TAM::tam.modelfit( mod1 )
summary(tmod1)

#############################################################################
# EXAMPLE 4: data.read | Comparison Q3 statistic
#############################################################################

library(sirt)
data(data.read, package="sirt")
dat <- data.read

#**** Model 1: 1PL model
mod1 <- TAM::tam.mml( dat )
summary(mod1)

#**** Model 2: 2PL model
mod2 <- TAM::tam.mml.2pl( dat )
summary(mod2)

#**** assess model fits
# Q3 based on posterior
fmod1 <- TAM::tam.modelfit(mod1)
fmod2 <- TAM::tam.modelfit(mod2)
# Q3 based on WLEs
q3_mod1 <- TAM::tam.Q3(mod1)
q3_mod2 <- TAM::tam.Q3(mod2)
summary(fmod1)
summary(fmod2)
summary(q3_mod1)
summary(q3_mod2)
# }