Validity and Reliability of Scores Obtained on Multiple-Choice Questions: Why Functioning Distractors Matter

Main Article Content

Syed Haris Ali Patrick A. Carr Kenneth G. Ruit

Abstract

Purpose Plausible distractors are important for accurate measurement of knowledge via multiple-choice questions (MCQs). This study demonstrates the impact of higher distractor functioning on validity and reliability of scores obtained on MCQs. Methods Free-response (FR) and MCQ versions of a neurohistology practice exam were given to four cohorts of Year 1 medical students. Consistently non-functioning multiple-choice distractors (<5% selection frequency) were replaced with those developed from incorrect responses on FR version of the items, followed by administration of the revised MCQ version to subsequent two cohorts. Validity was assessed by comparing an index of expected MCQ difficulty with an index of observed MCQ difficulty, while reliability was assessed via Cronbach’s alpha coefficient before and after replacement of consistently non-functioning distractors. Result Pre-intervention, effect size (Cohen’s d) of the difference between mean expected and observed MCQ difficulty indices was noted to be 0.4 – 0.59. Post-intervention, this difference reduced to 0.15 along with an increase in Cronbach’s alpha coefficient of scores obtained on MCQ version of the exam. Conclusion Multiple-choice distractors developed from incorrect responses on free-response version of the items enhance the validity and reliability of scores obtained on MCQs.

Downloads

Download data is not yet available.

Article Details

How to Cite
Ali, S. H., Carr, P., & Ruit, K. (2016). Validity and Reliability of Scores Obtained on Multiple-Choice Questions: Why Functioning Distractors Matter. Journal of the Scholarship of Teaching and Learning, 16(1), 1-14. https://doi.org/https://doi.org/10.14434/josotl.v16i1.19106
Section
Articles
Author Biographies

Syed Haris Ali, University of North Dakota School of Medicine and Health Sciences

Resident (postgraduate medical trainee), Dept. of Internal Medicine, University of North Dakota School of Medicine and Health Sciences

Patrick A. Carr, University of North Dakota School of Medicine and Health Sciences

Associate Professor, Dept. of Basic Sciences, Assistant Dean for Faculty Development and Director of Education Resources

Kenneth G. Ruit, University of North Dakota School of Medicine and Health Sciences

Associate Professor, Dept. of Basic Sciences and Associate Dean for Educational Administration and Faculty Affairs