Sensitivity of Achievement Estimation to Conditioning Model Misclassification

Abstract

Large-scale assessment programs such as NAEP, TIMSS, and PISA use a sophisticated assessment administration design called matrix sampling that minimizing the testing burden on individual students. Under a matrix sampling approach, not every examinee is administered every item, which poses currently intractable challenges to estimating individual achievement. Instead, population achievement is estimated via a latent regression approach that uses item responses and a vector of examinee background information (e.g., gender and ethnicity). This vector of covariates, used in an imputation model (more commonly called a conditioning model), is assumed to be fully measured, without error. Using simulated data that follows typical large-scale assessment designs, this paper provides some evidence that departures from this assumption can have a meaningful impact on conditioning model parameter estimates, subpopulation achievement estimates, and under- or over-estimates of subpopulation differences. Findings from this study indicate that the severity of parameter estimate bias depends on the measurement error mechanism and the generating ability distribution. Policy implications and impediments to detecting the impact of measurement error are briefly discussed.