Too cool for school ? : The effects of gamification in an advanced interdisciplinary course

Educators have been attracted to gamification because of its apparent appeal to students who are digital natives, but more research is needed to evaluate the effects of gamification on student motivation, learning, and related outcomes. This article presents the results of a gamification project conducted in an upper-division interdisciplinary course and suggests that gamification is not “too cool for school” but rather, an emerging instructional tool that will need to be filtered through the complexity of student experience in order to live up to its claims.

Understanding learning is a Gordian knot of a challenge, just the kind of intellectual exercise that tends to motivate academics.So, while there is no dearth of research and publication on learning (and teaching), whether or not we are any closer to unraveling the knot remains an open question.Certainly, we have done a better job understanding the knot itself.While learning is often associated with educational research (practice), the study has inexorably expanded to include physiological, psychological, physical, and socio-cultural-political perspectives, making the full study of learning to be very much a syncretic art.A recent contender for the attention of those interested in teaching and learning is gamification, which draws inspiration and momentum from sources not often associated with serious pedagogy (Educause, 2011).This study explores one example of the nexus between gamification and learning in an interdisciplinary college class.
Broadly speaking, the term gamification, coined in either 2005 or 2008 (depending on who you ask), refers to the use of various elements from games in non-game contexts (Deterding, Dixon, Khaled & Nacke, 2012).With origins in the technology industry, the concept quickly obtained a toehold in the business world, especially for marketing applications (Huotari & Hamari, 2012).To provide an idea of the scope of that infiltration, a 2011 Gartner report predicted that over half of all businesses would employ gamified marketing strategies by the year 2015 (Gartner, 2012) and M2 research predicts that gamification services will reach $2.8 billion in revenue by 2016 (McCormick, 2013).Some of the most commonly used examples of gamification are retail programs where participants accumulate extra savings or other rewards for continued patronage.Foursquare, Fango, and StackOverflow are popular gamification apps.
The widespread use of such techniques to engage customers in business began to catch the eye of educators who saw the potential inherent in engaging students.If the current generation of students spends a great deal of time playing online games, the reasoning goes, then why should we not harness that same kind of motivation to engage them in the classroom?Large funding agencies, such as the MacArthur and Bill and Melinda Gates Foundation, have begun pouring support into prototype projects that examine the application of gamification techniques at all levels of education (Landry, 2012).Advocates and pundits of gamification in education have been churning out articles, posts, and publications extolling its benefits and generating widespread publicity for successful pilot projects (Erenli, 2013;Jensen, 2012;Kapp, 2012;Smith, 2012;Yee, 2013).
If one were to see through the hype, however, questions have been raised about whether or not gamification can deliver on its early promises.Gamification often looks and sounds "sexy," in other words, highly appealing, particularly because it transforms educational opportunities into experiences that more closely resemble already popular games.That being said, most faculty do not view education as inherently a game and skeptical of so-called "edutainment."In other words, the attempted integration of gamification into higher education has produced an uneasy alliance, both philosophically and practically.A handful of studies have found issues with the depth of learning achieved through the application of gamification principles and have questioned the transfer of the primarily extrinsic rewards associated with gamification to deeper, intrinsic understandings (Deci et al., 2001;Donston-Miller, 2012;Dominguez, et al., 2013;Malone & Lepper, 1987).The emerging consensus seems to be that most students are enthusiastic to participate in gamification experiences, but whether or not their participation results in improved motivation for learning remains to be seen and the question forms the basis of the current study.To phrase it colloquially, the research at hand asks whether or not gamification is too cool for school.

Research Design and Findings
This research study was conducted on a single face-to-face class: an interdisciplinary, upper division course that is included in the general education requirements at a medium-sized, rural regional comprehensive university.This particular class was chosen because of the interdisciplinary nature of the subject matter, i.e. capitalism and because the classroom demographics were (very roughly) representative of the undergraduate student population as a whole.Of the forty-three students enrolled in the course, 54% were female and 46% male.Approximately twothirds of the students were upper classmen, listed as either juniors or seniors at the time of enrollment.Three were non-traditional students, ten of the students were registered in the Honors College and six were registered as varsity athletes.The class consisted of eighteen different majors, with highest numbers in history and education.In the spring of 2013, the instructor developed an extensive gamification assignment for the long-standing course, the former of which became the subject of the current research.The study asked consenting students to provide researchers with copies of all course materials related to the gamification exercise and to complete a short survey regarding their perceptions of learning through gamification.
Originally, the instructor considered a complete revision of the course design centered around gamification.Because of the paucity of research related to increased learning outcomes and gamification, the instructor chose a more conservative approach and to relegate the gamification concept to a single activity.In this case, the traditional end-of-the-semester review activity, as it was both cumulative, and thus representative of the course as a whole, and a selfdirected exercise that had built-in levels of success.Rather than being instructed to complete the entire review activity, students were incentivized by being offered tiers of accomplishment where the number of review questions completed resulted in progressively higher points.This tiered technique is adapted from the gamified concept of achievements (Habgood, 2007).
The literature on gamification indicates that the technical expertise to create high-end gamification experiences serves a barrier for adoption (NMC 2013).In the case of "The Capitalism Quest for Knowledge," the researchers chose a simple, low-tech text-based format.The ancestors of today's highly-detailed graphical games, text-based gaming has a long and rich history beginning with early offerings like Zork (1977) and Colossal Cave Adventure (1976) that utilized the format primarily to take advantage of the limited capabilities of early consumer computers.Players discovered the distance between "open program" and "open dungeon door" was short enough to be conducive to intuitive gameplay (Deitz, 2010).Additionally, the nature of text-based games's verbose descriptions in lieu of digital graphics remain very similar to the popular trope of "Choose-Your-Own-Adventure" stories among youth literature (Green & Jenkins, 2014).The choice to use a text-based format bypasses the need for developing complex graphics, but the approach still necessitates the use of a gamification platform for access, navigation, and interactivity.
Rather than engineer a home-grown, text-based gaming platform, the researchers employed a hybrid model.They identified an existing, free-to-use online text-based quest platform in combination with a series of databases created and maintained by the university.The platform allowed the narrative structure and non-linear navigation desired.The limitation of this platform was that it did not allow individual responses to be tracked, accumulated, or rewarded.This feature, that was necessary to provide the motivational structures intrinsic to gamification, which was enabled through use the associated databases.Even the use of existing technological solutions required considerable time, effort, and expertise to adapt to the gamification process, and, in the end, the blend was not perfect.Those interested in gamifying their instruction should consider the investment needed to provide appropriate technology and technological support.In addition to technological adaptation, the instructor took considerable time to redesign the assignment.Along with an instructional developer, the instructor applied key aspects of gamification, including nonlinearity, achievements, and a persistent narrative structure to the redesign (Elson, Breuer, Ivory, & Quandt, 2014).
To distance the assignment from its original roots as an in-class review exercise, and to inject an atmosphere of gamification, the assignment was made entirely nonlinear.In a traditional review exercise, students begin at "Question 1" and proceed with the assignment until completing "Question 34."While there are indeed linear games, a nonlinear structure provides more opportunity for players, or in this case forty-three university students, to interact with the assignment; input and response are some of the most basic gamification attributes that can be applied.In the new, gamified version of the review, rather than being presented as a linear progression, the questions were divided into five thematic clusters of roughly six questions each for a total of 34 possible quests.Students, who in this instance additionally serve as players, were able to determine their own progression path, rather than be influenced by the linearity of a traditional assignment; students could begin and end where they chose.
One of the most distinguishing features of educational gamification practices is the provision of intangible rewards (Hsu, Chang, & Lee, 2013).Often, these rewards take the form of valueless, collectible encouragements such as badges, points, or levels of achievement.For example, modern digital games often offer achievements for a variety of activities, from simply playing for the first time, to completing a challenging task.Additionally, points awarded can accumulate into tiered levels that serve as progressive milestones of accomplishment (Zuckerman, 2014).The researchers assigned specific questions in the original review point values commensurate with their relative difficulty and time investment required.For example, questions that required a brief, subjective response from the student/player were worth considerably fewer points than questions that required participants to produce larger creative works, to engage in higher-order thinking, or to leave the classroom on excursions around campus.Students did not have to answer all of the questions presented.They could choose to do more questions of lower point value or fewer questions with higher point value, or employ other strategies as they saw fit.
Framing the entire activity were a series of ranked achievements presented as the goal to which participants were to aspire.In order to align these levels of achievement with the course's theme of capitalism, they were given thematically appropriate designations, from "Apprentice" to "Maven."The Apprentice achievement required twenty points, while Maven was unlocked at sixty.Those three students who achieved the maximum possible points (61+) first were given an additional award.Grades can be seen as a form of tangible, extrinsic motivation.To reduce this factor, the instructor awarded all students who participated to at least the Apprentice level with a blanket participation grade, without further consideration for achievements earned.The researchers included this aspect of the gamified activity in order to observe student reaction to this gaming trope, and to evaluate the likelihood of motivation tied to intangible rewards.
In order to take advantage of gamification's concept of persistent narrative structure, and to more elegantly tie the activity to the major themes of this interdisciplinary course, the Capitalism Quest for Knowledge was couched in terms more common to games than in traditional review assignments: "questions" became "quests," "modules" of questions became "quest lobbies," each characterized with the name and visage of a luminary from the history of capitalism ("The Karl Marx Lobby," "The Adam Smith Lobby").The researchers chose language carefully to emphasize the participants' roles as "players" rather than "students," with particular focus on points, achievements, and quests.This ensured that the verbiage reflected the assignment's status as a participatory grade, rather than a graded evaluation.
Once the Captitalism Quest for Knowledge was built, it remained to be seen whether or not it would motivate the students.All forty-three students participated in the Quest for Knowledge in some way over the course of approximately two weeks, and all but one participated both inside and outside of class.Thirty-two students completed at least the minimal level of points (1-20) to receive full credit for participating in the activity.Approximately onethird achieved scores over 60 points, thereby receiving the rank of Maven.This level of participation, particularly that above and beyond the minimal requirements, seemed to represent a considerable increase in engagement over the traditional in-class review activity, which never engaged students to this degree or depth.
In studies of educational gamification, one of the primary assumptions made is that students will find gamified instruction attractive, because it mimics online games already popular with this generation of digital natives.As part of the Quest for Knowledge, the instructor surveyed students on prior experience with online gaming.56% of the responses indicated considerable experience with online games, from one responder who said he played "a lot, a lot a lot" to another who indicated earlier use as a small child.What was perhaps more surprising is that a significant portion of the responses (28%) indicated that they had not participated in online games prior to the exercise, citing them as "irrelevant", "pointless", and "a waste of time."Certainly not all of these students came to the exercise with an established fondness and familiarity with the structures and norms of gaming.
Gamification also rests on the assumption that competitive reward structures, such as those found in games, can fuel motivation.While the number of students who participated in the Quest for Capitalism suggests strong motivation, the survey responses are more skeptical and show a much more mixed response by students.The research survey (included as a "quest" for 1 point) asked two open-ended questions about student perceptions of gamification.The first question asked students to relate the reward structure to their motivation.

Motivation Assessment
The Quest for Knowledge is based on motivational practices from the concept of gamification.What effect did the use of badges, points, and prizes, have on your motivation to participate?
On the positive side, students most often stated that the competitive nature of the exercise was a strong motivator.On the other hand, the negative responses suggested both that other rewards (money, grades, peer pressure, success in major) were equal to or more powerful at motivating them to learn.Motivation is a key attribute of gamification, but one that requires further exploration in the higher education environment (Hamari, 2013).
Critics of gamification have suggested that student engagement is superficial-it may seem cool to do, in other words, but it does not positively affect learning outcomes.The second survey question asked students about their perceptions of the connection between gamification and learning outcomes, in this case, a review of the course material prior to the final examination.Despite their skepticism about gamification and motivation, the students participating in this study did perceive that the exercise positively affected their learning (90%).
Students indicated that the quests pushed them to apply the knowledge that they had obtained in the course in new ways, particularly in those quests that required them to teach or communicate ideas to others.As one respondent stated, " Very effective.It makes me think about what I learned in class and apply it to my everyday life.Very cool."Other researchers have shown, however, that student perception of learning is not a reliable yardstick for assessment of actual learning outcomes (Clauss & Gedey, 2010).In other words, what students learn and what students think they learn can be two very different animals.Perception of learning has suffered in the literature to the point that some editors do not accept scholarship of teaching and learning studies based only on learner perception, even with validated instruments (DiPiro, 2010).Because the Quest for Knowledge was a review exercise, the degree of student learning from the exercise itself, as distinct from the cumulative knowledge of the non-gamified remainder of the course, is particularly difficult to disentangle.That being said, the researchers did discern patterns to the responses that shed some light on the quest to reach learning outcomes.
One of the criticisms of gamified instruction is that it is better suited for lower-level learning outcomes, such as basic understanding, than for higher-level learning outcomes, such as those related to critical evaluation or creative thought (Dominguez, et al., 2013).The activities designed for Quest for Knowledge were intentionally created to provide a continuum of learning outcomes, and so it is possible to track how many students chose to do those quests associated with lower-order thinking and those with higher-order thinking, as articulated by the Bloom taxonomy.
Interestingly, the students seemed to respond roughly equally to the lower and higher order thinking questions.The instructor did not grade individual quests, but, in spite of this fact, most students participated seriously and provided meaningful responses to all of those quests in which they participated.From the instructor's perspective, Quest 11, in which students had to draw, or otherwise create, an allegory for their ideological perspective on the course themes, was perhaps the most challenging.Not only did nine students tackle this question, they did so at a level of depth and diversity that well exceeded expectations.It would appear from this pattern of responses that students did not shy away from higher-order thinking when choosing their Quests for Knowledge.
The pattern of choices the student players made is perhaps more reflective of other contextual factors.While Quest 1, a simple application exercise called "Who's Speaking," received the highest number of responses, it should be noted this quest was also the first quest to appear on each student's screen.In general, those questions with an experiential component, such as Quest 2a, which required students to leave the classroom and take photos that displayed positive and negative aspects of capitalism, or Quest 27, which required students to purchase fast food, tended to draw higher numbers than those requiring specialized skills, such as Quest 30 which involved quantitative analysis, or the availability of specialized materials, such as Quest 10, which involved contributing a slogan to a "graffiti wall" located in the physical classroom.Students also struggled with questions that required them to form groups, such as Quest 12, which involved setting up a race to complete a structure built from plastic bricks, or Quest 9, which required students to get others to respond to an online poll that they created.Because students completed most of their quests outside of the classroom and all hours of the day, this made the process, perhaps by necessity, much more solitary than it had been as a fully classroom-based exercise.This correlates with the literature on self-paced or competency-based pedagogy, which calls for the need to recreate community outside of the classroom in order to compensate for declining social interaction within it and with the gaming literature that emphasizes the need for social structures in gaming (Hamari & Koivisto, 2013;Petrovic, 2012).In some ways, the students on the Capitalism Quest for Knowledge re-created their own community using social media.Several students created, on their own volition, a Facebook page for the course, which included links, discussions, news postings, and more.On that page, they discussed the Quest for Knowledge extensively with each other (over 150 posts), evaluating its benefits and drawbacks, and providing the instructor with a set of recommendations for the next iteration of the exercise.

Discussion and Implications
One, and perhaps the only, central conclusion that seems to characterize all pedagogical studies is that certain teaching methods or approaches may be better suited for particular disciplinary, institutional, demographic, or individual contexts.In this case, the success of even a limited run of gamified instruction may be due, at least in part, to the interdisciplinary nature of the course.As a subject of study, capitalism is inherently interdisciplinary, integrating elements of history, economics, sociology, cultural studies, ethics, political science, law, and other fields (Muller, 2003).And that integration is proving to be increasingly popular, and courses in the history of capitalism are on the rise across the United States (Schuessler, 2013).
Interdisciplinary studies, by definition, do not "lay claim to a universally-recognized core of knowledge" but rather seek to draw on disciplinary knowledge while at the time "transcending it via integration" (Repko, 2012).For the gamified course, integration of the diverse disciplinary perspectives is achieved through the persistent narrative structure.Further, the fact that this course is not embedded in a particular disciplinary structure, makes it especially conducive to divergent, constructivist, and non-linear outcomes, also often associated with interdisciplinary teaching (Davies & Devlin, 2006).Whether or not this association with interdisciplinary studies helps or hurts the potential adoption of the pedagogy of gamification is not clear, however, as interdisciplinarity continues to find considerable resistance in U.S. higher education, though there is some evidence that this resistance is waning and more institutions are looking to adopt interdisciplinary approaches, particularly in the general education curriculum.
The researchers gleaned some evidence that the interdisciplinary nature of the course contributed to the success of the Capitalism Quest for Knowledge.In their survey responses, students highlighted the ability to navigate their own paths through the quests, which allowed them to choose those quests that fit best with their major, their interests, or their skills.Some questions or themes were more conducive to those with business majors, for example, while others were more suited to those in the social sciences.With no grade expectation, the exercise allowed students to experiment with perspectives outside of their comfort zone, and to engage in ways of thinking that were new to them, a factor several students highlighted as beneficial.This plasticity may also help to explain the variation in choice of quests.In some ways, it could be said that the gamified quest structure transformed what can be a liability in upper division general education courses, i.e the array of majors and background knowledge, into a strength.Additionally, the survey results showed little to no evidence of student resistance to extreme changes in instructional delivery, a factor often cited in other studies of radical redesigns (Nijhuis, 2005).This is perhaps attributable to the interdisciplinary mixture in the course, so that students did not have a single pedagogical convention from which to form a basis of resistance.
Gamification advocates highlight increased engagement as one of the primary benefits of the approach.Students play online games for hours, and gamification aims to tap into that same deep and persistent motivation.In the case of this class, the findings suggest that this assumption may be an overgeneralization.Most students did participate eagerly in the exercise, but several students, declined to pursue the quests fully.Interestingly, three of these non-participants were among the highest-performing students in the class otherwise, and they could afford to not receive the participation grade associated with the exercise.Each cited lack of interest, particularly in the competitive aspect, as their reason for non-participation.The fourth student who declined to participate simply chose other priorities at the end of the semester.Several students (four) also participated in only the most superficial way, providing only cursory responses, often not even tied to the quest.Across the board, these were all students who had been consistently under-performing throughout the semester.On a more positive note, several other under-performing students showed renewed interest in the course and its material through the exercise and became some of its highest scorers, so the effect was not consistent across the board.These results do seem to be consistent with those associated with other forms of selfdirected or competency-based learning (Liaw, 2008;Reeboy & Semb, 1991).Despite the ambiguity of some of these results, the researchers do intent for next iteration of this course to include more, not fewer, gamified elements.

Table 1
Quest for Knowledge: Final Rankings

Table 3
Question 2: How effective was the Quest for Knowledge in bringing together themes of the course for you?

Table 4
Quest for Knowledge Activities Ranked by Bloom's Taxomomy