Examining the Connection between Classroom Technology and Student Engagement

Pedagogical strategies increasing rely on technology as a way to address challenges facing the contemporary higher education classroom. This study investigated the use of a tool designed with the specific purpose of student engagement in a large classroom. The tool allows students to post to a community based discussion in a manner similar to a Twitter feed. Despite engaging in a community dialogue, results show that as usage of the technology went up a student’s sense of community learning went down. This result prompted the authors to consider how this tool was utilized in the classroom, and while implementation diverged from the original design intention the usage mirrored common technological teaching tools like a threaded discussion board. We close with a warning that adoption of a novel technology alone does not produce a greater sense of community learning.


Introduction
The "state of higher education" is a topic of increasing concern in the United States.Example talking points include declining performance of U.S. students in the core areas of science, technology, engineering, and mathematics, and state budget shortfalls that result in sweeping education cuts.In this often politicized and highly turbulent era, some have argued that the golden age of U.S. higher education is over (Wolin, 2012).Those less critical, when pressed, do acknowledge a general, "decline of higher education" (Goldrick-Rab, 2012).However, despite the changing landscape the National Center for Education Statistics (NCES) report that undergraduate, "Enrollment in degree-granting institutions increased by 11 percent between 1990 and 2000.Between 2000 and 2010, enrollment increased 37 percent, from 15.3 million to 21.0 million" (Snyder & Dillow, 2011).
NCES also reports that there are approximately 3 million graduate students enrolled in programs across the United States including public, private, and first-professional programs.This is an increase of approximately 51% over the last 30 years (Snyder & Dillow, 2011).The rhetoric regarding a "decline of higher education" seems aimed at performance outcomes and not at rates of attendance.
Individual classroom sizes have increased in concert with growing attendance and a currently depressed hiring cycle (Biemiller, 2011;Williams, 2012).As a result, instructors are teaching larger classes (Williams-June, 2009;Wolff, 2012).Technology is touted as a way to overcome the challenges associated with heavy teaching loads and large class sizes, providing instructors with the ability to engage students in their courses.This paper briefly describes the rationale for integrating technology into the higher education classroom, presents the results of a study conducted by the authors regarding the interaction between classroom technology and student perception of the classroom environment, and discusses the study results in the context of best practices for pedagogical reform and educational technology in higher education.

Literature Review: Teaching with Technology
Research into best practices for teaching and learning in higher education indicate that effective teaching encourages students to take an active role in their learning through cooperation with their peers to meet learning goals (Peel, 2005;Carnell, 2007).Best practices also suggest that both the instructor and student need to respect multiple perspectives and ways of learning (Chickering & Gamson, 1999).Large classes present a unique set of challenges as instructors seek to create this engaged learning atmosphere.The use of technologically oriented pedagogy is becoming a "go to" method as instructors attempt to connect with students in large and small classrooms alike (Koeber, 2005).Research shows that when technology is used appropriately it can enhance student learning (Kuh & Vesper, 2001), providing a platform supporting student engagement, active learning, and cooperation (Thomas & Kuh, 2005).Many academicians view the use of technology as the future of higher education pedagogy for classrooms of all sizes and emphases (Hughes, 2007;Berrett, 2012;Young, 2012).In turn, students increasingly expect to be engaged by technology (Benson, Ore, & Haney, 2002).
Pedagogical strategies that rely on technology seek to satisfy four general goals 1) engaging students, 2) increasing learning, 3) increasing performance, and/or 4) improving classroom management/efficiency.Each goal relates to the other, but is not necessarily equivalent.Instructors may adopt a strategy to achieve one outcome and not the others.In the best practices scenario, an instructor desires the combined result of all four outcomes.Consider as an example the use of electronic response systems, or "clickers" (Persell, Pfeiffer, & Syed 2008).Clickers are a popular pedagogical tool (Mollborn & Hoekstra, 2010).Despite the popularity, it is possible to use a clicker to achieve one instructional goal at the expense of others.Having students use clickers makes it easy for them to respond quickly, efficiently, and with minimal self-identification to a variety of classroom related activities.Despite the ease of classroom management, a clicker does not inherently guarantee a higher level of learning.While this statement seems like common sense, the rhetoric and packaging accompanying many of these new technological teaching tools suggests that usage alone will increase student engagement and learning.As one set of instructional design technicians at our university remarked, "[our tools] bolster engagement inside and outside the classroom..." Statements like this are not uncommon as pedagogy in higher education continues to adopt novel technology used both inside and outside of the classroom.Email and physical in-boxes are increasingly crammed with solicitations promoting hardware, software, and techniques oriented toward greater and greater usage of technological pedagogy.

Research Questions
The challenges facing higher education have prompted instructors to turn to technological pedagogy as a way to address the contemporary demands of teaching.Likewise, students increasingly expect to be engaged by the use of technology as part of the learning process.However, does the use of novel technology positively correlate with student engagement?This is a timely question to answer given the emphasis on technology centered pedagogy in higher education.If the rhetoric and packaging of new technology are to be believed, usage alone should supplement the learning process in a way that makes the classroom experience studentcentered.Proponents of technologically driven pedagogy take the argument further to suggest that the engaging student environment will result in deeper learning and higher performance (i.e., better course grades, better learning of content, and even higher retention rates).The pressing needs of contemporary higher education rely heavily on technology to achieve these goals.The research question driving this paper can simply be articulated, "Does the use of technology correspond with a greater sense of student engagement in the classroom?"This study focuses on the use of one specific teaching tool that was developed at our university.This tool is part of a package that recently won a prestigious award at the 2012 Campus Technology Innovators Awards under the category for teaching and learning.The website for this tool describes it as follows, "[this tool], a social networking-powered mobile Web application, creates a collaborative classroom, allowing students to provide near real-time feedback during class and enabling professors to adjust the course content and improve the learning experience.Students can post messages to [this tool] using their Facebook or Twitter accounts, sending text messages, or logging in to the [tool] Web site."Interaction with the tool is similar to a Twitter feed.Students are able to post real-time flowing discussion in a space using hash tags and content management.The idea is to allow for an open forum of student-driven discussion fostering emergent and synergistic topics that help guide instruction.A feed can be displayed in real time on a projector during a class session.It can also be used outside of class to continue discussion.A tagline for this tool describes it as bridging the gap between instructor and student.The line reads, "Opening the back channel in large lectures."

Research Setting: Course Redesign
Following a five-year study conducted by The National Center for Academic Transformation (NCAT) institutions across the U.S. are examining the way course content is taught to students.Our university is engaged in a similar effort.As part of our campus redesign initiative, data are being gathered about student perceptions of engagement as well as adoption of novel technology in redesigned courses.These data make it possible to conduct a quantitative analysis of the research question guiding this study.One of the central themes of redesign initiatives currently sweeping across the U.S. places substantial emphasis on the use of technology as a part of course redesign.In fact, NCAT says there are two main goals for course redesign and that each relates to the use of technology in the classroom.The first goal is to improve the quality of teaching, and the second to reduce the costs of higher education.Their website emphasizes technology by saying, "Course redesign using information technology is key to achieving both outcomes" (Transformation, 2012).The emphasis on technology integration as a part of pedagogical reform is found on the campus where the current study takes place, making this setting an ideal place to address the research question proposed.

Student Engagement
Student perceptions of the learning environment are often studied in concert with other variables commonly associated with student success because of a demonstrated linkage between student engagement and academic integration and persistence (Kuh et al., 2008).The definition for "student engagement" varies among these studies, but the Scholarship of Teaching and Learning generally agrees that student engagement is defined by active participation in lessons through discussions between student and instructor, discussions among students, and includes an element of student ownership of course material in the learning process (Fraser, 2012;Harper & Quaye, 2010;Wolters & Taylor, 2012).A survey protocol known as the Classroom Experience Questionnaire (CEQ) was created on our campus to assess student perception of the classroom learning environment, in particular, student perceptions of active and collaborative learning (for more on the creation of this survey protocol see Morris et al., Forthcoming).The CEQ assessment tool was developed based on previous empirical work (Piburn et al., 2000) on course redesign using instruments like the Reformed Teaching Observation Protocol as a model (MacIsaac & Falconer, 2001).
An important goal of creating the CEQ protocol was the development of a valid and reliable instrument for assessing course reform using student self-reports of their experiences in the classroom, rather than those of third-party observers (MacIsaac & Falconer, 2002).As such, the CEQ was developed by modifying assessment items that appear on existing protocols making questions more easily interpretable by a diverse group of college students in an online survey format.This process involved a "deconstruction" of currently established measurement methods, translating measurement items into classroom practices that embodied common learning environment constructs such as "student engagement" (Sawada et al., 2002).
Data used for this study come from the course redesign project on our campus using the CEQ student self-report database.When this manuscript was prepared the database represented seven courses and 11 separate sections (to date the database has grown more than six-fold).As a part of the redesign one course adopted the novel technology designed to "open the back channel in a large lecture."This course is the focus of this study.

Methods
Results come from a single section of an introductory Political Science course, largely made up of first and second year students.The total course roster had 187 students.Using blinded ID numbers, students were randomly selected to participate in the online CEQ survey.No exclusion criterion was used to disqualify students from participation in the study.A total of 74 students (40%) volunteered and completed surveys.Of those, 11 students did not complete the survey after clicking on the link to begin, these empty or missing cases were dropped leaving 63 valid cases available for analysis.
Our first step determined if the CEQ instrument produced similar constructs as are present on established measurement protocols.Dimension reduction followed techniques of Confirmatory Factor Analysis (CFA) using Principle Axis Factoring (PAF).Dimension reduction included a coefficient of alienation of ≥ .30 to eliminate weak relationships.Cronbach's alpha was used as a measure of the internal reliability for each construct.Following dimension reduction and internal reliability analysis, CEQ items were used to create Bartlett (DiStefano, Zhu, & Mindrila, 2009) factor regression scale scores of student engagement.Scales were modeled with data related to the technological tool under investigation.Based on the crosssectional nature of these data we have also reverse order tested the models.To account for the small sample size (N = 63) our final step ran a post-hoc power analysis.

Predictors and Outcome
The dependent variable tested relates to usage of the technology under investigation; usage of this tool lent itself to a count analysis.Simply put, usage can be counted because usage is equal to the number of unique postings logged into the system by each student (i.e., the technology is used in a manner similar to a Twitter feed).According to the redesign emphasis on technologically driven pedagogy (coinciding with developer selling points) frequency of posting should have a positive correlation with self-reported engagement.
Measurement of student engagement came from Likert scaled (from 5 -Strongly Agree to 1 -Strongly Disagree, with 3 -Undecided) CEQ items that were transformed into factor regression scores.We created factor scores to ascertain the correlational relationship between student engagement and usage of the novel technology adopted as a part of course redesign.The analysis we ran did not include additional control variables because there is no theoretical reason for us to believe that typical controls like age or gender significantly influence posting. 3

Hypothesis
Previous research has shown that use of technology in the classroom produces positive results especially with respect to students' perceptions of teaching effectiveness (Pippert & Moore, 1999), active collaborative learning (Hung & Yuen, 2010;Junco, 2011), sense of engagement with course content (Persell, 2004;Pearson, 2010), academic activities (Thomas & Kuh, 2005), and increased academic achievement (Wright & Lawson, 2005).Thus, our hypothesis is: the CEQ constructs will be positively correlated with usage of technology in the classroom, as a student's number of posts increases their perceptions of engagement in the course will also increase. 4

Results
Based on the design of the CEQ these data should produce three main factor structures for classroom culture.Existing definitions of student engagement include an emphasis on active participation in lessons through discussions-between student and instructor and between students-as well as a measure of student ownership of course material.Grounded in these ideas we name the three CEQ constructs: 1 -Learning Community (LC), 2 -Constructivist Pedagogy (CP), and 3 -Equity (EQ).Tables below present the questions used to create each construct.Table 1 displays the results of the first CFA showing a pattern matrix grouping around the LC construct.To augment the findings of the CFA we also included a measure of Cronbach's alpha.
Results of Table 1 show that items making up LC achieved necessary sampling strength to be modeled as a unique construct (KMO Bartlett test statistic = .753).Table 1 also shows that LC is a strong construct with good factor loadings (.585 to .895) and possesses strong internal reliability (Cronbach's alpha = .829).Table 2 displays the results of the CFA for CP.
3 This should not be read as suggesting that usage of technology is unrelated to age or gender.These data also lack demographic variables such as race or nationality of student which could potentially prove to be confounding variables.This is a known limitation of these data.A further confounding variable is a person's inclination (including self-efficacy beliefs) toward usage of novel technology.However, because usage of this tool was a requirement of the class typical controls were inappropriate.More detail on how the technology was used as a requirement of the course is presented below. 4The hypothesis presented is based on previous research but also reflects the contemporary belief that usage of technology in the classroom is inherently beneficial to student learning.Results of Table 2 show that items making up LC achieved necessary sampling strength to be modeled as a unique construct (KMO Bartlett test statistic = .824).Table 2 also shows that CP is a strong construct with good factor loadings (.668 to .912) with strong internal reliability (Cronbach's alpha = .906).Table 3 displays the results of the CFA for EQ.The EQ construct with three items had moderate to low loadings, failed the KMO and Bartlett test (.449) and also had low Cronbach's alpha (.41).CEQ data did not adequately produce the EQ scale construct, and the EQ scale was dropped during the remainder of the analysis.
Table 4 presents the three variables that were modeled (plus a modified version of LC) to address our research questions.Table 5 presents zero-order correlations between the three variables.Negative correlation was present between both measures of student engagement and usage of the technology, with a statistically significant relationship between LC and Posts.As posts increased LC decreased by .399,P = .019(see Table 5, Column 1 -Row 2).This finding wholly rejects our hypothesis.To dig deeper into the relationships between student engagement and usage of the technology a zero inflated Poisson model with Posts as the outcome was run.We decided on a zero inflated model based on the correlation results of Table 5 and the presence of Poisson errors, but primarily due to the large number of 0's in these data.A Vuong test compared the zero-inflated model to the standard Poisson model and determined if the outcome should be treated as zero inflated.Results of the test showed that a standard Poisson model was a better fit for these data (Z = 1.43,P = .0770).We also used the Stata command "vce robust" to account for a small violation of the assumption that the distribution variance equals the mean by obtaining robust standard error estimates (Cameron and Trivedi, 2009).Model 1 in Table 6 presents the Poisson regression model showing that for a one unit increase in the respondents LC score, holding the other variables constant, the difference in the log of expected posts would be expected to decrease by .476(P =.011).Meaning, as the odds of posting increased a student's sense of engagement related to the learning community significantly decreased.CP was not a significant predictor for usage of the technology.Our hypothesis assuming that students who utilize a technology designed to engage students by "open[ing] the back channel" of communication was not supported.Both correlation and regression modeling found a negative relationship between usage and sense of engagement.
The next step was to reverse order test the model to see if Posts would be equally predictive of students' sense of engagement.Based on the zero-order correlation LC was chosen as the dependent variable, with Posts and CP predicting.Table 4 displays descriptive statistics for a transformed version of LC.Transformation was necessary based on the skewness and kurtosis of these data points.We transformed the LC factor scores by adding a constant and then taking the square root.This modified variable produced the results of LC Transformed, reported in Table 4.The transformed LC was well within acceptable levels to be modeled with Ordinary Least Square (OLS) regression.Model 2 in Table 6 displays the results of this OLS regression.Again we found the strong relationship between LC and CP (negative as a result of data transformation), but specifying LC as the outcome as predicted by Posts produced no statistically significant result.The better model for these data is Model 1.
The final step was a post-hoc power analysis.A critical z value of 1.64 with 63 observations resulted in the power (1-β err prob) .62.The LC produced a Z value of |1.53|.These results suggest that these data have a moderate ability to detect significant results.McFadden's R 2 of .169 in Model 1 should be interpreted with caution given the Poisson distribution; however, this model accounts for nearly 20% of the variation of posting with only two predictors modeled.Given the strong result produced primarily by the LC (β = -.476,P = .011),these findings are worthy of consideration.

Discussion
The total picture of our analysis rejects our initial hypothesis.A student's posting is correlated with their sense of engagement as measured by the CEQ, but in a negative direction.Additionally, we find that the appropriate ordering places a student's sense of engagement antecedent to posting.For this course, the learning atmosphere determines the amount of student usage on a technology designed to promote student engagement.
Stretched to a maximum, and admittedly a generalized claim is thin, the findings of this study cast doubt on adopting technology as the answer for achieving student engagement.More precisely, these findings indicate that usage of an award winning engagement tool is not enough to create a greater sense of engagement under the conditions adopted in this course.In interpreting these results, it may be helpful to consider what we mean by "adoption" or "integration" of technology in the classroom.Researchers examining information technologies in higher education have found that one of the most popular adoptions of technology into the higher education classroom is for course management (general goal 4 presented above), rather than as an aid to achieve learning objectives (Selwyn, 2007).Additionally, researchers have noted that instructors who adopt technology in their classrooms typically do not change their teaching styles to incorporate the collaborative potential of these technologies (Grasha & Yangarber-Hicks, 2000).Breen (2001) has found that the use of technology in higher education classrooms is highly variable across institutions and classrooms within institutions.In our case, further investigation into how this technology was implemented was needed to further interpret our results.
Thus, after running these analyses and discovering the negative correlation we sought additional information from the course instructor regarding the implementation of technology in the course examined, a course that this instructor has taught for many years.As a part of the redesign process the instructor had been trained in the usage of this tool.The instructor also had access to a professional development community including support staff from campus units overseeing teaching and information technology.
Upon request, it was possible to obtain a copy of the syllabus associated with the class.The focus of the course described in the syllabus states, "The course offers an understanding of the forces that influence the behavior of individuals and institutions in and around government."The social science orientation of the class is sufficiently similar to the previous research settings that provided a basis for our hypothesis (cf.Howard, 2005;Koeber, 2005;Little, Titarenko, & Bergelson, 2005;Clark-Ibáñez & Scott, 2008;Koeber & Wright, 2008;Hill, Arford, Lubitow, & Smollin, 2012;Hoop, 2012).Content related to American government, associated social structures, and individual behavior clearly has potential fodder for student engagement.The next question is then: how was the technology utilized?According to the syllabus, usage of the technology was a part of the "course requirements."The following is the instructor's expectation for use of the technology: "Even though this is a fairly large class, I will strive to learn all your names… At the beginning of the semester students were given a brief introduction to the tool during class and were then expected to use the tool based on course expectations.This is an adaptation of the originally intended usage.Recall that this tool was designed to provide an outlet for free-flowing thought so that unanticipated/emergent themes could inform the progression of a course.The implementation under review adapts the usage of this tool such that the instructor controls the topics that were to be discussed.Clearly not all adaptation of a tool improves the usage of that tool.If the instructor had allowed students to freely discuss the course material, students may have been able to interact as a community, and this may have positively influenced their perceptions of the learning environment.
However, instructors are advised to utilize technology as a tool for achieving their own goals for student learning.The CIO over information technology on our campus said the following regarding pedagogical adaptations, "For example, when we released [this same tool], our classroom discussion tool, we had our own ideas about how a faculty member might use it.But we've found that faculty are using it to engage students in ways that never occurred to us.When people invest their own creativity into a technology to make it better, that's a great sign that the technology truly is innovative" (Thomas, 2012).It would seem that there are limitations to adaptation.According to the CIO's definition of "innovation" technologically-driven pedagogy should be implemented to engage students.In the case of these data and findings, it could be argued that innovation has failed and not technology per se.However, because these data are cross-sectional, lacking a control that uses the technology in the originally intended manner it is not possible to answer this question directly.
The findings raise additional questions.Looking at how the instructor required use of technology, the requirements seem like a reasonable adaptation, especially if a classroom goal is to get students to engage with the subject matter in a novel way.The adapted usage is similar to a threaded forum discussion or blog, another popular technological course augmentation.So when is adaptation appropriate or inappropriate?Controlling the flow of conversation is an adaptation of the originally intended use for this tool but is this change the antecedent for our findings?
It is just as reasonable to assume that those who use the technology most often do so because they feel isolated from their peers in class and this isolation is what drives them to use the technology as a way to engage.5 Isolation also speaks to the model ordering we found.Students in this class who use the technology most often feel less engaged in the classroom despite their frequent use of the technology.Whatever the explanation, it is clear that usage of this technology in this way for this class has failed to create a more engaged learning environment.Usage alone does not produce a community experience, as was expected by the instructor following their participation in a course redesign program focused on making learning more engaging and student-centered through technological pedagogy.
During the process of reviewing the redesign data being tracked on our campus the outcome discussed here was uncovered, and upon finding the negative correlation we began drafting this manuscript.As higher education continues to adopt more technologically driven pedagogical strategies we felt it was important to use this result as a reminder to pause and consider how a tool will be used before classroom adoption.Research has shown that technology can improve the student experience.However, implementation of any pedagogytechnologically oriented, or otherwise-should be done carefully and thoughtfully.Findings support previous research showing that the goals an instructor has for student learning outcomes should drive the use of any teaching technique and not the opposite (Cuban, 2009).This also has implications for course redesign efforts currently sweeping across the U.S. Putting redesign pedagogies before learning outcome development renders the redesign ineffectual.No matter how attractive a new tool may be, if its use does not support a learning outcome it can potentially have a negative impact on community based and student-centered learning (see Blackie, Case, & Jawitz, 2010 for more on student-centered learning).

Limitations and Suggestions for Future Research
The major limitations of this study have already been mentioned.This study followed one class for one semester.Ideally, this study would have included a control class utilizing this technology in the originally intended manner and for added strength a class not utilizing this technology at all.These data were collected to assess the effectiveness of the redesign efforts on this campus.The specific details or the ideal control scenarios needed to make definitive conclusions about the usage of the technology under investigation were not a part of the original research design.Despite these limitations, a significant outcome was uncovered within these data that should not be ignored.These data show that students who perceived a greater sense of community in this class used the technology less than peers who used the technology more often.Another potential explanation for this outcome is that students who utilized the technology frequently are high academic achievers, typically going above and beyond course requirements.Frequency of posts would then be correlated with student achievement, showing that those who post often do so primarily because it is a classroom expectation.Future data analyses will need to address this possibility.
As the trend in higher education continues to adopt technological pedagogies it is vital to pause and remember that usage of technology alone does not produce more community.As educators, the authors of this paper argue that this finding should serve both as a warning and encouragement.The warning seems clear: adoption of technologically driven pedagogy should be done to achieve previously articulated goals for student learning outcomes and not simply for the sake of implementation.The encouragement is also intuitive, though potentially less obvious: as students continue to demand engagement via technology, as more courses move to the Internet for their "classroom," and generally as future generations of students rely more heavily on technology as a medium for learning, some have feared a diminishing role for the formal educator (Healy, 1998;McCain & Jukes, 2001).These findings should put to rest some of these fears.As courses evolve to include more technology, the need for skilled teachers will not be replaced.In fact, the growing size of the pedagogical menu will increasingly require thoughtful and careful implementation of technology to achieve the desired learning outcomes.
Morris, R.C., & Parker, L.C. Journal of Teaching and Learning with Technology, Vol. 3, No. 1, June 2014.jotlt.indiana.edu 2 Morris, R.C., & Parker, L.C. Journal of Teaching and Learning with Technology, Vol. 3, No. 1, June 2014.jotlt.indiana.edu6 Morris, R.C., & Parker, L.C. Journal of Teaching and Learning with Technology, Vol. 3, No. 1, June 2014.jotlt.indiana.edu8 -provided opportunities for students to challenge opinions expressed in class..675Item 2 -encouraged students to participate actively in class..729Item 3 -provided opportunities for students to ask questions..895Item 4 -allowed students to answer a question or solve a problem in more than one way..654Item 5 -maintained a climate of respect within the class for what others had to say..585n = 5 items | Cronbach's α = .829 *These items were scored on a scale from 1 to 10, 1 representing complete student choice to 10 instructor choice.Morris, R.C., & Parker, L.C. Journal of Teaching and Learning with Technology, Vol. 3, No. 1, June 2014.jotlt.indiana.edu7

Table 6
Poisson Regression Estimates of Posts & OLS Regression Estimates of Learning Community must stress that participation means much more than just showing up! …After most class sessions, I will post a brief question or comment on the [name of technology] page.At least twice during the semester, you should post a response.(Note that even if you post anonymously, I will know who you are.)" Journal of Teaching and Learning with Technology, Vol. 3, No. 1, June 2014.jotlt.indiana.edu10