Diverse Student Perceptions of Active Learning in a Large Enrollment STEM Course

The concept of active learning as a superior mode of instruction has recently received great attention in the education research literature. It holds promise of steering students away from rote memorization towards higher order thinking (as defined by Bloom’s taxonomy revised). However, few studies focus on student perceptions of higher order thinking activities and diverse student voices are all but absent in this regard. This study applies a combined approach of exploratory qualitative and supplementary quantitative analysis to address this gap. We examined perceptions of underrepresented and non-underrepresented students regarding their engagement in active learning to foster higher order thinking. The study was set within a large enrollment (198 students), undergraduate course in the area of science, technology, engineering and mathematics (STEM). The study sample comprised of 33 purposefully selected, ethnically and racially diverse students. Data sources included class attendance/participation, graded activity assignments, and a perception survey. Class attendance and graded assignments were used to triangulate responses on the perception surveys. The Generic Inductive Approach supported our qualitative analysis. Quantitative data were analyzed via two-way ANOVA, non-parametric Mann-Whitney Test (when assumptions did not hold) and simple linear regression. Findings include three themes that cut across groups; participants perceived their higher order thinking skills improved, that there were benefits and challenges to active learning and a fear of failing the course. Quantitative data from the active learning activities and attendance supported similar engagement and achievement in higher order thinking activities across race/ethnicity groups as differences failed to reach the a priori established significance threshold. This study extends the knowledge on active learning and demonstrates that it was possible to engage underrepresented and non-underrepresented students equally and effectively in higher order thinking actives in large enrollment courses and that students perceived this as beneficial.

in decreased satisfaction (Eichler & Peeples, 2016;Gutwill-Wise, 2001). In addition, there is a lack of institutional commitment to developing, implementing and supporting research based teaching across a wide range of STEM courses (Wieman, 2007). Lastly, there are simple physical impediments. Many undergraduate STEM classes take place in large lecture halls constraining instructional practices through room architecture and seating arrangements (Baldwin, 2009). This is of course extremely problematic because research show that lecture only approach is not effective for most students (Freeman et al., 2014;Wieman, 2007;Zoller, 1993), and is particularly ineffective for underrepresented students in large lecture classes with a reputation for high failure rates (Haak et al., 2011;Hrabowski, 2011;Mervis, 2010;Mulligan, 2000). Underrepresented and underserved students in college include Black, Hispanic, Native American and Pacific Islanders as well as first generation college students. These groups are less likely to complete four-year degree programs as White and Asian peers (US Department of Education, 2014). In addition, students in high failure courses are dissatisfied with the quality of STEM teaching, particularly when size limits student-teacher dialogue (Baldwin, 2009). Improving the persistence of underrepresented college students typically focus on advising, scholarships and tutoring with limited studies examining classroom based strategies (Winkelmes, Bernacki, Butler, Zochowski, Golanics & Weavil, 2016). Although limited, a study by Winkelmes (2013) found transparency regarding how college students learn, what they learn and why a course is structured a particular way "showed promise for improving underserved students' educational experiences" (p.1). Another study found that students perceived "technology-nested" instructional strategies as a strong component to content engagement and enjoyment (Lumpkin, Achen & Dodd, 2015 p. 12). These studies highlight the promise of explicit instruction or transparency and using technology in college courses for all students.
While active learning is more effective, problems can arise when student's perception of active leaning techniques become negative over time as the novelty wears off over the course of a semester (McDougall, 2013;Nguyen et al., 2016;Cooper, Downing, Brownell, 2018). The limited discussion of student perception in the literature reports that students perceive in-class activity as effective but did not specifically assess perceptions of higher order thinking (Nail, 2012;Nguyen et al., 2016). Certain types of active learning strategies such as using case studies and creative activities can be applied to pertinent situations to support higher order thinking (Bean, 2011). However, students in active learning environments resented the "intellectual effort" needed for successful completion of activities (Smith & Cardaciotto, 2012). Evidently, activities targeting higher order thinking require the most intellectual effort. This presents a stark contrast to the standard lecture format which encourages students to be mostly passive learners (National Research Council, 2003) who rely heavily on the lower levels of thinking such as memorization of facts or formulas (Baldwin, 2009;Brainard, 2007;National Research Council, 2003). Whether there are differences in the perception of active learning to support higher order thinking among underrepresented students is currently unknown.
Therefore, the purpose of this study was to examine both underrepresented and nonunderrepresented student perceptions of collaborative, active learning activities that support higher order thinking in a large enrollment STEM course. The overarching research questions guiding this study are: "What are underrepresented and non-underrepresented students perceptions of active learning activities within a large lecture course?", "What are underrepresented and nonunderrepresented student perceptions of their higher order thinking skills after engaging in the course?" and "Are there differences in these perceptions?".

Methods
This exploratory mixed methods study (Creswell et al., 2003) was drawn from a larger IRB approved research project. This study consisted of a sample of 33 students from a large (198 students), upper division (juniors and seniors) undergraduate, health science course at a major public university located on the west coast of the United States during Spring 2016. The sampled participants included 19 females and 14 males. Age ranged from 20-29 years. The sample was ethnically and racially diverse. Eleven Asian, Hispanic and White students were purposefully selected (Patton, 2015) as they represented the larger racial makeup of the course. 1 We focused on the three largest racial/ethnic groups in the course. Students self-identified their race/ethnicity and then were randomly selected within each group. Table 1 illustrates student identification in greater detail as well as other demographic information.

Course Description
The course for this study was Measurement and Evaluation in Kinesiology (ENS 305). This course is required for all emphases within the Kinesiology major, the largest major at the university (1,800 students, >5% of total undergrad enrollment) and part of the second largest field for undergraduate degrees conferred in the United States (National Center for Education Statistics, 2016). Other common names of this major include Exercise Science, Physical Education, Exercise and Sport Science, Health and Exercise Science, Exercise Science and Wellness, Exercise and Fitness, Kinesiological Sciences, and Exercise Physiology (Boone, 2000). This course was chosen to be redesigned with a focus on technology and active collaborative learning to support student achievement. In previous semesters 20% of grades fell into the repeatable category. It has recently been the main bottleneck course leading to delays for students trying to advance through the major, impacting four year and six year graduation rates. The course builds on prerequisite statistics courses by using descriptive and inferential statistics to examine the quality of assessment tools, study designs and inferential analysis used by Exercise Science and Kinesiology professionals. The emphasis of the class is on the active use of higher order thinking skills to apply assessment principles for the determination of the quality of assessment tools and usefulness of the data for meaningful decision-making. The concepts of higher order thinking according to boom's taxonomy were explained to students on the first day of class (the only day with mandatory attendance), and students were reminded and made aware of what type of thinking they were expected to perform for the activities. The course met twice a week for one hour and 15 minutes each session.
Five graduate assistants (GAs) helped facilitate the active learning in class environment. In order to optimally guide deliberate practice during the active learning exercises and maximally engage student (particularly those who may be struggling) feedback, encouragement and supervision are crucial. To achieve this in a large class more than one person is necessary (Park, 2004). Activities that require higher order thinking and complex problem solving (i.e. "when the going gets tough") will have to be supported by accessible expert feedback (Harland, 2003). GAs were trained by the second author specifically to improve student engagement and higher order thinking. Training consistent of three meetings before the semester to familiarize GAs with Blooms taxonomy (revised), key aspects of active learning and its effectiveness compared to standard lecture formats as well as practice activities. During the semester, GAs and instructor met on Friday of each week to go over events and activities of the past week, the class material for next week, selected activities for the week and practiced every activity as intended for class before implementation. GAs acquired specific content expertise necessary to assist in the course, prepared active learning activities and provided analysis and evaluation of their effectiveness (i.e., reflection) (Aronson, 2011).

Data Sources and Data Analysis
Data sources included class attendance/participation as assessed by wireless audience response system (i>clicker2); 12 graded assignments of higher order thinking activities (worth 1 point each) completed in flexible groups; and a researcher developed, open ended perception survey (see Appendix 3). The students were randomly assigned to groups and stayed within that group for the duration a given topic was covered (i.e., concept of validity) which could span between 1-3 days and were reassigned to new groups for the next topic. Group assignments were based on student identifiers (i.e., last 4 digits of student ID) and seating charts projected on the screen. The survey was administered in class at the end of the semester. Response rate was 89% (177 out of 198 students). Class attendance and graded assignments were used to triangulate responses on the perception surveys (Patton, 2015).
The qualitative open-ended survey data was analyzed using an inductive approach (Charmaz, 2006). To analyze the survey data, we organized participant response using the qualitative data software NVivo (QSR, 2015). Using part of the question itself as a starting category (e.g., "What are your perceptions of active learning?" became the category "Student perception of Active Learning"), we organized open-ended student comments. Next, these open-ended comments were coded inductively, moving from more concrete 'open and focus' codes to more abstract 'theme' development (Charmaz, 2006). From this process, three themes emerged from the data: improved higher order thinking skills; benefits and challenges of active learning in large enrollment courses; and fear of failing.
Quantitative data (i.e., student points for activities and attendance) were analyzed to assess differences among race/ethnicities and sample groups via three univariate Analysis of Variance followed by post-hoc pairwise comparison with LSD adjustment and two sided independent sample t-tests, respectively. Assumption of normality was checked with Q-Q plots and the Shapiro-Wilk test. Assumption of homogeneity of error variances was assessed with Levene's test. If the assumptions did not hold logarithmic transformation was attempted, or non-parametric analysis of multiple groups (i.e., races/ethnicities) were assessed with the Kruskal-Wallis Test and between groups comparisons were assessed with the Mann-Whitney Test. Simple regression of attendance on class session was used to assess attendance over time and to identify differences among race/ ethnicities by comparing 95% confidence intervals of unstandardized coefficients. Adjustment of αlevel for multiple comparison was done by the Bonferroni method were applicable. Level of significance was set a priori at α = .05 for all analysis.
This exploratory mixed-methods study showed that across the three race/ethnicity groups, all students perceived that their higher order thinking improved, perceived both benefits and challenges of active learning in large enrollment courses and that throughout the process, they had a fear of failing (see appendix 1).

Improved higher order thinking skills
Although participants were juniors and seniors, many indicated that this course was the first time they were explicitly taught higher order thinking skills. Participants stated "No [they've never heard of higher order thinking]"; or "Never learned the details of higher order thinking." One student stated "I've never heard of higher order thinking. I'm used to standard lectures". These statements shed light on the lack of explicit attention to higher order thinking strategies and activities these students received in previous courses, regardless of ethnic background. Providing explicit instruction in higher order thinking gives students the tools they need to work in groups in meaningful ways. Because students were taught explicitly, most students regardless of ethnicity, perceived their higher order thinking skills had improved. For the question, "Do you believe your higher order thinking skills have improved? Please explain." One student stated: "I have applied these forms of higher order thinking skills in other classes this semester." Another participant said, "I do [feel that my higher order thinking improved]." "I did feel like I applied the knowledge I acquired towards assignments." Finally, a participant commented that working with others helped. The student stated, "[My higher order thinking skills improved], through partner activities and engaged learning." There were no discernable differences among the Race/ethnicity groups as all perceived they improved their higher order thinking skills in a similar manner and magnitude. This was supported by quantitative assessment of points obtained during the on-line class higher order thinking activities.

Results
Journal of the Scholarship of Teaching and Learning, Vol. 20, No. 1, April 2020. josotl.indiana.edu 46 Participants also describe benefits and challenges of engaging in higher order thinking skills through active learning activities within a large enrollment course. They stated that some of the challenges included working with unprepared students, not enough time and lack of feedback due to class size. One participant stated, "The size of the class makes getting deeper into concepts and materials difficult." This sentiment was expressed across several students and race/ethnicity in the study. One self-identified Asian male was particularly concerned about active learning in a large lecture setting. He stated, "I understand the purpose of this teaching method, but I believe that to have college students who have been taught another way their entire lives is difficult for us to shift gears. I learn better through the other way of teaching that we are used to. Yes, some students may "memorize" materials without fully understanding but others like myself, memorize and understand at the same time." This student clearly explains the struggle of trying to learn in a different course format. Further, despite the course's focus on Bloom's taxonomy, he believes that "understanding" course material is sufficient.
Participants also felt that at times the course was unorganized because there were so many students moving around getting into "active learning groups." Students were not familiar with moving around and engaging in group work in most of their courses, so it took some getting used to. One student commented: "At times, active learning activities in large lecture hall got a little chaotic and unorganized but I think it's worth the trouble." These sentiments about the benefits and challenges of active learning activities within a large enrollment course were shared across race/ ethnicity groups, and this was again supported by quantitative analysis such as engagement with the course as measured by attendance. Attendance was generally very high (mean > 84% for all groups, Figure 2). Nevertheless, even at this high level, there was still a significant difference among groups (i.e., no ceiling effect, χ2 (2, N = 75), p = 0.15) with a mean rank of 28 for Whites, 47 for Hispanic/ Latinos and 37 for Asians. There was no significant decline in attendance over the course of the semester for either Whites or Hispanic/Latinos group (b = -0.2 to -0.5, t(24) = -1.30 to -1.43, p = . 165-.207) but attendance did decline for Asians (b = -1.0, t(24) = -2.80, p =.010).On course session 23, attendance dropped for all students and particularly so for Asians. However, there was no significant difference across trends for each group as indicated by overlapping confidence intervals (-1.20 to 0.22, -0.62 to 0.14 and -1.83 to -0.27 for Whites, Hispanic/Latinos, and Asians, respectively).

Fear of Failure
The subject of points and how they negatively or positively affect grades came up quite often in these surveys. The active learning activities were completed for extra credit points. Students had a choice whether or not to complete these activities. In this way, students were encouraged to make their own decisions about their level of engagement. For these assignments, participants were concerned about being in "weak" groups, or groups they had to "carry." One participant stated, "I felt I had "carry" my classmates who did not show up or participate." Another student stated, "I didn't like working and depending on random people for some of the in-class assignments." All participants in this study completed the activities but their reason for doing so mostly focused on the outcome, not the process. Some students stated that they appreciated the effort. "[The professor] is trying hard to change how students learn but that is hard to do when we have been "taught" to learn a certain way for 20+ years." Another student expressed frustration with never having a "right" answer. She stated, "I didn't like how some questions were answered by the professor with another question or with multiple questions. There was no clear answer." Because of these issues, they were also afraid of failing the class/not passing the final. One student spoke to this, "I just hope I did well enough. Unfortunately, GPA is extremely important for grad school, so I just need an "A" for the class. I will not be evaluated on "what I learned in [this course]." The education system is built this way." All race/ethnicity groups shared this fear and expressed equal concern about their grade.

Discussion
This exploratory mixed-methods study examined underrepresented and non-underrepresented student perceptions of active learning in a large enrollment STEM course. It illustrates the complexity between integrating best practices in large enrollment STEM courses and the need for high grades so that all students have an opportunity to succeed in post-secondary education. The qualitative data indicated that the majority of participants across groups perceived active learning as beneficial but challenging in a large class, that their understanding and engagement in higher order thinking skills improved, and that they were very concerned about failing the course. The perceptions of improved higher order thinking skills were shared across all race/ethnicity groups, and the quantitative analysis supports this notion. All race/ethnicities showed similar achievement in higher order thinking activities, demonstrating wide applicability of this approach. This is of note because while lecture only approaches are ineffective for most students (Freeman et al., 2014;Wieman, 2007;Zoller, 1993), they are particularly ineffective for diverse students in large lecture classes with a reputation for high failure rates such as is often the case in many STEM courses (Haak et al., 2011;Hrabowski, 2011;Mervis, 2010). Thus, this approach provides an avenue for all students to engage in cognitively challenging concepts in critically meaningful ways. While it was beyond the scope of this study to objectively assess higher order thinking skills with a standardized, validated instrument, this outcome is nevertheless highly encouraging because students are capable of accurately reporting their own learning (Chesebro & McCroskey, 2000;Smith & Cardaciotto, 2012). In addition, their perceptions have the potential to influence learning outcomes (Lizzio et al., 2002;Smith & Cardaciotto, 2012). Indeed, students that perceive that their higher order thinking skills are improving are more likely to persist in engaging the skill than those that do not believe they are improving.
Other studies have also shown that, across all groups, purposeful and persistent practice of higher order thinking enhances these skills, the disposition to use them and their self-confidence (Miri et al., 2007;Nelson & Crow, 2014;Nguyen, 2016). In addition to similar achievement, engagement was also consistent across race/ethnicity with students from all assessed groups showing high levels of engagement (as assessed by attendance) with Hispanic/Latino students attending most regularly. As mentioned in the results, there was a noticeable but none significant drop in attendance during course session 23.This day was announced to be a specific "question and answer" session and did not involve activities for points which likely lowered attendance and participation rates. This is meaningful because it suggests that students may have been trained to attend class only if it affects their grade. This would indicate low internal motivation for attending class. If this data point is removed, then there is no more significant change in attendance over the course at the adjusted α-level. The qualitative data also supports this finding as students across race/ ethnicity stated their frustration with the point system for the extra credit active learning activities. Students engaged in the extra credit activities to boost their grades because academic incentive structures value numerical indicators of improvement more than qualitative indicators. This is an important consideration given that problems can arise after initial implementation of active learning strategies when over the course of a week student perceptions become less favorable (McDougall, 2013). Based on the attendance data we did not observe such a tendency. This was in spite of the fact that students usually resent having to expend intellectual effort (Smith & Cardaciotto, 2012) without consistent, tangible reinforcements.

Implications and Recommendations for Practice
The current study extends the knowledge of active learning and examines the effects of best practice into a large, upper level undergraduate course with underrepresented students. A proper college education must go beyond mere knowledge acquisition and skill development in a specific discipline. This notion is not just philosophical but supported by stakeholders such as employers who expect students to gain proficiencies in non-material specific areas such as team work and higher order thinking (Hart Research Associates, 2015). In fact, a recent study of over 400 employers who hire a large percentages of college graduates indicates that more employers (25 vs. 15%) believe that long term career success depends on general knowledge and skills rather than knowledge and skills of a specific field or position (Hart Research Associates, 2015). This is contrary to the popular tendencies for colleges and university to emphasis 'job readiness,' and career focused/ specific training (Brand & Valent, 2013;Docherty, 2014;Fein, 2017). The highest valued learning outcomes by these employers include oral and written communication skills, teamwork, decision making, critical thinking and application of knowledge, analysis, evaluating information, innovation/ creativity numeracy and solving problems with people from different backgrounds (Hart Research Associates, 2015). One cannot help but notice that this list is almost entirely opposite to standard lecture based education in large classes where isolated, individual information memorization is the predominant style of learning and passivity abounds (Baldwin, 2009;Brainard, 2007;National Research Council, 2003). Despite the obvious antithesis of a standard, large, lecture course format to these desired learning outcomes STEM education is still dominated by this type of instruction (Stains, 2018). Also of note is that these desired learning outcomes are heavily weighted towards the higher rungs of Bloom's taxonomy of higher order thinking.

Recommendations regarding 'Improved higher order thinking skills'
Going forward, one can build on the positive perceptions expressed by students regarding engagement in active learning for higher order thinking. One should highlight the importance of these skills and how to achieve them. We provided students with detailed explanations on what these skills are using Bloom's taxonomy. We also highlighted how these skills are valued by key stakeholders such as employers. We emphasized higher order thinking as a targeted student learning outcome for the course as well as outlined which specifics skills where targeted with each activity.

Recommendations regarding 'Benefits and Challenges in large enrollment course'
The challenges expressed by students are difficult to overcome. Students were randomly assigned to groups as research has shown the benefits of heterogeneous groups, but most of the participants felt this was problematic. Problems such as students being unprepared or unwilling to contribute sufficiently to group activities were commonly expressed. Strategies to rectify this issue include assessments prior to engaging students in activities. Readiness assurance testing such as employed in team based learning formats offer one such option, and several approaches have been described (Antoun, J., Nasr, R., & Zgheib, N. K., 2015). Individual contributions to collaborative group activities are difficult to monitor in large classroom environments. Peer evaluation is likely the most feasible solution to this issue. Recently, a combined analytical mapping approach requiring minimal computational effort has been developed which allows for reliable individual grade assignment based on peer marks (Dijkstra et al., 2016;Spartar et al., 2015). We aim to investigate the incorporation of these strategies in future interactions of similar courses.
Further challenges such as the unfamiliarity of active learning may actually present a desirable learning opportunity. A certain level of uncomfortableness will likely be unavoidable when employing active learning for higher order thinking. Requiring student to step out of their comfort zone and engage in an unfamiliar activity is an important learning outcome in and of itself. Ideally, this can be done without introducing undue anxiety (Cooper, 2018). However, one should be cognizant and empathetic to student perceptions by being transparent in expectations and explicit in direction. Further, it is likely prudent to first engage in activities that more heavily rely on the lower levels of Bloom's to carefully scaffold the progression to higher levels in order to aid in the transition to the new format and way of thinking (Vygotsky, 1980). Organizational issues perceived by students could be resolved with a simple training session at the beginning of the course. Group assignments were based on student identifiers and seating charts projected on the screen. Students were expected to easily transition to new groups at the beginning of each class. We did not anticipate that this would present a challenge to upper division students. In the future, a simple training sessions where students will be asked to find their groups quickly and efficiently several times with several different arrangements could likely alleviate much of this issue. We can envision a 'fun challenge' where students successfully try to improve the time it takes to arrange themselves into the appropriate groups based on varying the overhead display. After a few such practice runs, we would expect students will become highly efficient in the desired transition.

Recommendations regarding 'Fear of Failure'
The issue of fear of failing is a difficult one to resolve. It was not dependent on students fearing that they were unable to learn from the activities. Rather, it is to students worrying that they would not be rewarded with a high grade. An obvious solution would be to remove all points associated with the activities, but this may be counterproductive. Participation seemed to (in part) depend on the points attached to the activities. Therefore assigning fewer or no points would likely result in reduced participation. On the other hand, increasing the points allocated to the activities could further increase uncertainty and anxiety as this was already the case with just assigning extra credit points. A potential solution could be to allow more space for 'failure' while keeping incentives for participation. One such strategy is to only count a certain number of activities towards the course grade (e.g., the ten activities with the highest scores). As students do not know beforehand which activities they will get graded on and how high their score will be, this should reduce the stakes and still provide an incentive for participation. Concomitantly, being assigned to a 'weak' or poorly performing group would hopefully be perceived less impactful as activities from such days will likely not factor into the top ten scores. Lastly, perceived discomfort of not having clear cut, single solution answers is again an issue that is understandable but likely unavoidable. Clear cut, single solutions are nice and comforting but simply do not represent the real world, especially not for issues that require higher order thinking. While one should be aware of and empathize with students concerns, one should not shy away from exposing them to uncertainty or a multitude of possible solutions to any given problem if one cares to prepare students for more than taking simple examinations.

Conclusion
Active learning specifically targeted to higher order thinking was equally effective in engaging underrepresented as non-underrepresented students in this large enrollment STEM course (i.e., Measurement and Evaluation in Kinesiology). It also improved students' self-assessed higher order thinking skills. Clear definitions of higher order thinking and transparency regarding focus, importance, and impact of these skills were perceived as important elements for successful implementation by these students. Several challenges were perceived equally amongst students from all backgrounds regarding peer performance in group activities, organization difficulty due to class size, unfamiliarity of active learning approaches and grade pressures. Strategies addressing these challenges such as peer evaluation, organizational practice, progressive transition of activities and alternative grading strategies present promising targets for future investigations.

Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.