IRT and SVD: Implementing Psychometric Methods in New and Complex Situations

by Quinn Nathaniel Lathrop

Institution: University of Notre Dame
Department: Psychology
Degree: PhD
Year: 2015
Keywords: Item response theory; psychometrics; singular value decomposition; data mining; educational data; missing data; computation statistics
Record ID: 2057928
Full text PDF: http://etd.nd.edu/ETD-db/theses/available/etd-04132015-134701/


Most psychometric techniques used to analyze assessment data are designed to work with complete data. The rapid increase in the availability and power of technology has contributed to the growing use of computerized tests and related methods. The data arising from these new and complex situations challenge traditional psychometric techniques because of their size (as there is much more data) and their vast missingness (as students respond to only a small subset of possible items). This dissertation focuses on the effect of missing data on psychometric techniques. When individuals respond to different items of varying difficulty, the psychometric techniques that rely on complete data can perform poorly. This dissertation proposes using Singular Value Decomposition (SVD), a matrix decomposition technique often seen in data mining, as a psychometric tool. The major result is that SVD is a viable psychometric tool that appears largely robust to missing data and to the missing mechanism. This document provides analytical and empirical justification for SVD's use with psychometric data under missing data. Chapter 1 introduces relevant IRT techniques such as nonparametric IRT and a nonparametric item fit statistic. Then, in Chapter 2, SVD is introduced and as well as an Alternating-Least-Squares (ALS) algorithm that extends the decomposition to missing data. Chapter 3 investigates the large sample properties of using SVD with psychometric data. SVD is shown to be a consistent ordinal estimator of student ability and a consistent ordinal estimator of item easiness. Chapter 4 presents simulation results that show that when students respond to different items of varying difficulty, whether the missingness is related to their ability or not, SVD can rank the students better than proportion correct and can better estimate the true relationship between student ability and the probability of a correct response. When missingness is related to student ability, SVD can rank the students, in most conditions, better than a parametric IRT model, even when the parametric model is correctly specified.