An Evaluation of Undergraduate Advisors Experience Using Learning Analytics to Support First-year Students

Abstract

Higher education institutions are now serving post-traditional students. With the ever-increasing diversity and complex needs of these post-traditionals, institutions are striving to design policies, programs, and institutional supports to best support their diverse needs. Many are venturing into the world of learning analytics to gain deeper insights into the student academic experience and leveraging data to improve student success and retention. Previous research has centered on the institutional level impact of learning analytics on student success and rarely gives representation to the experience of specific individual sub-groups of organizational stakeholders. This summative evaluation sought to capture the experiences of 5 undergraduate advisors who participated in a three-year pilot of Civitas Inspire, a learning analytics system, to support first- year students. The Comprehensive Mixed Methods Participatory Evaluation model served as a conceptual framework allowing for an in-depth exploration of advisors’ perspectives on six evaluation components: acceptability, social validity, program integrity, program outcomes, implementer competence, sustainability, and institutionalization. An examination of previous research identified capacity building, data integrity, messaging, and privacy/ethics as common challenges faced by institutions who have adopted learning analytics systems. Evaluation results found advisors encountered similar challenges. Prominent throughout the advisors narrative was the effects of shadow-culture on technology adoption efforts. Advisors expressed the need for greater stakeholder inclusivity; for institutions to acknowledge and understand stakeholder workflow, and the necessity for a connect the dots approach towards institutionalization efforts.