RELATIVE UTILITY OF THREE MODELS FOR USER EVALUATIONS OF LEARNING MANAGEMENT SYSTEMS: A HIGHER-ED INSTITUTION DECISION CONTEXT

Abstract

Learning management systems (LMSs) have broad adoption in higher education. Many institutions are re-evaluating their LMS strategy, comparing the relative advantage of alternative systems to the institution’s current LMS. As part of a selection/decision process, most institutions gather instructor and student user evaluation after a level of “hands-on” user experience. Gathering users’ input underscores their importance and relevance to the institution’s technology decision process. Reviewing publically available LMS evaluation reports, one finds no mention of any framework or model used to guide the user evaluations, though common concepts occur frequently across the various reports. Conversely, there is a rich tradition of technology acceptance and success research, with several competing information system (IS) models having been developed over the past couple decades. This paper explicates the three most frequently used IS models used in LMS academic studies – the Technology Acceptance Model, the IS Success Model, and the Task-Performance Chain Model. This study examined the relative utility of the three models to begin to build a bridge between the academic literature and the LMS selection/decision processes underway at many institutions. Utilizing existing data from an LMS selection/decision process at a large, public higher education institution, a qualitative analysis was conducted by coding student user responses to open-ended survey question using the respective models’ constructs. Quantitative analysis was conducted by first mapping closed-ended survey questions to the models’ constructs and then analyzing the data with each model using partial least squares structural equation modeling (PLS-SEM). Results indicated that technology acceptance and success models have considerable potential utility for understanding user input in the LMS selection/decision context in higher education. All three models exhibited utility in analyzing the student user evaluations from an LMS pilot and contributed to the foundation for building a proposed Learning Management System – Pilot Model (LMS-PM) that could provide for practitioners common nomenclature and a framework for understanding and sharing LMS pilot evaluation results.