Competent or not ? : Exploring adaptions to the neo-behaviorist paradigm in a sport marketing course

Educators and administrators are exploring competency-based education as an effective and efficient method to facilitate student learning. This reinforces a burgeoning neo-behaviorist movement in higher education which seeks to synthesize such behaviorist approaches with the cognitive focus of the last 20 years. The current research examines the outcomes in three years of a sport marketing class that blends cognitive-based and competency-based pedagogy. The first half of the course is primarily self-paced, with regular quizzes checking student mastery, while the second half of the course has students work in teams on marketing-related projects; a final examination assesses overall student learning. The research revealed that the blended approach resulted in complementary strategies which partially addressed the conventional criticisms of both cognitiveand competency-based pedagogical approaches. The study used paired sample t-tests to compare results on quizzes versus the final exam (N=111 students comprising 36,787 total student-responses), finding that the course's hybrid structure develops student learning in both lower-order and higher-order thinking (as per Bloom's taxonomy).

Competent or not?While this question may appear simple on the surface, it belies the potentially disruptive challenge that competency-based pedagogy presents to the current framework of higher education.As of 2015, the approach seems poised to expand its hold over higher education, particularly as pressure mounts to utilize increasing effective and efficient methods to facilitate student learning (Kamenetz, 2013).This predicted ascendancy is based on more than prognostication as an increasingly number of indicators suggest that a paradigm shift in higher education is taking place and that resources are likely to follow (Fain, 2014a(Fain, , 2014b)).Because of this growing attention, many disciplines are asking themselves how a competency-based approach might integrate into their signature pedagogies (Boyer, Edmondson, Artis, & Fleming, 2014).
The fundamentals of competency-based pedagogy, e.g., self-paced instruction (Roberts, Fulton, & Semb, 1988;Roberts & Semb, 1989, 1990), learning laboratories, and multiple choice testing, should seem familiar to senior faculty because it closely resembles the personalized system of instruction (PSI) pioneered by Fred Keller and colleagues (Keller, 1968),which came to the fore in the 1970s.Keller grounded his plan in the theories and applications of behaviorist psychology, which posited the centrality of observable actions as indications of learning.Those same disparity in the prior knowledge that the students had at the onset of the course.To address this issue, the instructor chose to adopt a competency-based approach and developed self-paced modules for the students to master in the first major section of the course.In these modules, students completed the readings at their own pace and were given the option to come to office hours with the professor to discuss questions on content.When students felt confident in the material, they took a unit quiz.
The instructor carefully designed the unit quizzes to follow best practices in competencybased pedagogy.Each quiz was delivered through the Learning Management System (LMS), in a computer lab setting.Each unit quiz consisted of five multiple choice questions, randomly selected from a bank of questions dedicated to that particular unit.Upon completing each quiz, students saw which questions they got wrong, but not what the right answers were; this was done to encourage students to return to the material to gain clarity on a particular topic (i.e., learn from mistakes) rather than just remembering answers to specific questions.After leaving the computer lab for the day, students no longer had access to the questions.Students could retake each quiz once, with rules that limited the total number of retakes and prevented students from retaking a quiz after moving on to subsequent units.
The instructor designed self-paced modules around the expected learning outcomes from the course (e.g., evaluating sponsorship activation strategies, developing marketing plans for realworld situations).In principle, correct answers confirm students' mastery, while incorrect answers on the unit quizzes show students where they may be deficient.This provides feedback to students that they can use, if they so choose, to return to the material or meet with the instructor to clarify points of confusion.When students see the same questions on the final exam as they saw during the quizzes, the expectation is that students will get the same ones correct as before, and students will be more likely to answer questions correctly that they had gotten wrong the first time.In other words, the assumption is that the iterative process inherent in the self-paced structure will lead to increases in student learning, as evinced by increased final examination scores.
To test this hypothesis, researchers gathered quiz and exam responses from students over a three-year period.All courses were taught by the same instructor and the readings and question banks remained static.The questions on the final exam were the same each year.The average class size was 38 students.Students were all junior or senior level undergraduates, 79% of whom were majoring or minoring in sport management; of the non-sport management students, 83% were majoring in another program within the College of Business.Data from 114 unique students and 269 unique questions yielded 36,787 total student-responses, or approximately 323 questions per student.
Using this data set, the researchers compared each student's performance on the quizzes to the final exam on a question-by-question basis.All questions that were seen twice by the student were included (i.e., any question the student saw on a quiz and again on the final exam).Quiz questions that did not appear on the final exam were excluded, and final exam questions that the student did not first see on a quiz were excluded.While all students saw the same final exam questions, the quiz questions seen were different for each student.
The initial quiz questions provide formative feedback to students about potential areas of deficiency, but they also offer the researchers a benchmark against which students' mastery on the final exam can be compared.Examination of these results allows us to see where student learning has taken place between the quizzes and the final exam, a necessary component in evaluating the effectiveness of the hybrid course structure.It is unrealistic to expect all students to earn 100% on the final exam, thus the focus must be on improvement from time A (the quizzes) to time B (the final exam) rather than just the final outcome.Table 1 shows the three possible performance outcomes when seeing a question multiple times (no change, gain, or loss), as well as the frequency with which each outcome was observed in the data.In the "No change" condition, we see that 83% of students who got a question correct on the quiz also got that question correct on the final exam; that is, students learned the information initially and that knowledge was retain/reinforced between the quizzes and the final exam.On the flip side, over half (55%) of students who got a question wrong on a quiz also got that question wrong on the exam (i.e., no learning occurred).Of greater interest to the researchers are the students who showed a difference in performance (a "gain" or a "loss") between the quizzes and final exam.Specifically, it was encouraging to see that in 45% of cases, students who got a question wrong on a quiz got that question correct on the final exam ("Gain") but disheartening to have 17% of cases where students got a question right on a quiz and wrong the final exam ("Loss").A chisquare goodness-of-fit test was conducted on these categorical data to determine if there was a statistically significant difference between Gain and Loss outcomes.That is, are students more likely to improve than to regress between the quizzes and the final exam?The analysis revealed no statistically significant difference in the distribution of Gains and Losses at the 0.05 significance level (χ 2 (1) =3.06, p = 0.08), indicating that students appeared as likely to improve as they were to regress.Thus, while seeing a large percentage of students showing Wrong-to-Correct and Correctto-Correct was encouraging, the results were clearly very mixed and the answer to the question of the impact of the self-paced intervention remained ambiguous.Self-paced learning may have characterized the first half of the course, but in the second half, the instructor chose a project-based design related to the creative application of the competencies mastered in the first half.This pedagogical approach has a long history in the cognitive literature, and has even spawned sub-branches of inquiry, including team-based and cooperative learning (Michaelsen, Knight, & Fink, 2002;Millis & Cottell, 1997).In this class, managed groups of 5-6 students each worked with clients to produce deliverables that addressed a marketing-related issue for the client.Past projects included conducting market research for a local minor league baseball team, preparing a sponsorship inventory portfolio for the county youth baseball league, analyzing social media activity around a major televised sporting event, or building the promotional calendar for the university's softball team.
The projects varied within each class; that is, each group in a semester worked on a different project.Students had approximately five weeks to compile the deliverables.In the sixth week, the last week of class, students presented their projects to the rest of the class and submitted their deliverables to the clients.Each project focused on different aspects of marketing and students were matched to their projects based on interest.Based on his knowledge of project-based learning, the instructor expected the students who chose a project in a particular field to have higher results on final examination questions on related topics, a result of greater understanding generated through the project process.
To test this hypothesis, the lead researcher (also the course instructor) coded each group project using the topics covered in course units; for example, a project on launching a product in a new market was coded as "Strategy," while one involving surveying fans was coded as "Market Research."Again looking at Gains versus Losses (see Table 3), researchers examined whether a sameness between the question topic and the topic of the student project resulted in better student performance.That is, is a student more likely to improve than regress when the final exam question relates to the topic of his or her project?A chi-square test for independence returned χ 2 (1)=0.316,p = 0.574, telling us that there is no statistically significant difference between Gains and Losses regardless of whether the project topic matches the question topic.The results caused the instructor to question his own competency as it would appear that there was no link between gains in mastery/learning and the subject matter of the projects selected.
Because this was a new pedagogy, however, the researchers chose to ask a different kind of question from the data.Self-paced learning, by its very definition, is a solitary endeavor and a diminished social learning environment is a common criticism leveled at competency-based learning models.Rather than correlating project themes with final examination questions, the researchers chose to consider whether or not the social learning involved in project based learning led to an increase in overall student learning.Once the researchers discarded the content categories, they found that the project experience did result in gains in student learning overall, particularly when comparing the results of the final examination to earlier results, rather than contrasting specific questions to each other.
To re-assess overall performance, researchers used a paired t-test (two-tailed) to compare each student's average quiz grade to that student's average grade on the final exam.Where students saw the same question on an initial quiz and a retake quiz, only the retake grade was used.This created a higher quiz score (the average student increase when counting the most recent score was 1.61% greater than when counting just the initial results), but was appropriate in order to isolate the effectiveness of the project as a learning intervention.
The dataset was checked for outliers and normality, 3 and a paired t-test showed a 2.16% increase in student performance, t(110)=4.07(p<0.0001). 4Researchers thus accepted that there is statistically significant improvement between the quizzes and the final exam, an improvement researchers attributed to the learning that took place while working on the group projects.
It seemed possible that gains between the quizzes and the final exam might reflect a test learning effect (cf.Little, Bjork, Bjork, & Angello, 2012); that is, that students were learning the right answers to the questions themselves as opposed to the content being assessed by those questions.This seems unlikely for several reasons.First, students were not given the right answers to questions they got wrong, preventing them from simply remembering which answer went to each question.Second, the large number of total questions (each student took an average of 142 questions during the unit quizzes) made it unlikely that students could remember the questions, especially given the 7-17 week gap between when the quizzes were taken and final exam date.Finally, as shown above, students were about as likely to improve on a question as they were to regress between the quiz and the final exam.A strong test learning effect should prevent that regression.Another possibility is that the students are simply getting more accustomed to the instructor and question-writing style of the assessments.If that were the case, one would expect to see a performance increase throughout the quiz-taking sessions, but none was found. 5Therefore, 3 Outliers in the dependent variable, that is, differences between the two groups, should be detected to ensure the validity of analysis (Warner, 2008).In addition, the distribution on the differences between the two related groups should be checked for normality before conducting a paired t-test.A Box Plot and Q-Q Plot revealed two outliers, but the Shapiro-Wilk test for normality (S-W=0.995,p=0.973) indicated that the differences are normally distributed.Even though these outliers did not result in a violation of normality, researchers removed them from the dataset to ensure the validity of data.Later analysis revealed a third outlier, which was also removed in order to maintain a consistent sample across all analyses.The differences in the new dataset maintained a normal distribution (S-W=0.988,p=0.441) with no outliers.

4
A paired t-test conducted on the original dataset, where the differences were normally distributed but there were outliers, also showed the same 2.16% increase in student performance from the quizzes to the final exam, t(113)=3.94(p<0.0001).Thus, removing the outliers had no effect on our conclusions.

5
To check for this, a least squares regression line was calculated for each student's scores, where a positive slope would indicate an upward trend through the 23 quiz sessions.A one-sample t-test was conducted to compare the mean of the slopes (M= -.00047) for the regression lines with population mean (µ= 0), which showed no statistically significant change in student performance over time, t(110)= -0.832 (p=0.407).
while the test learning effect cannot completely be ruled out, its effect appears to be negligible and would not explain the significant gains seen in the data.
Inspired by these new perspectives, the researchers turned their attention to the overall course design.The literature suggests that there are valid reasons why the predecessors of competency-based pedagogy had disappeared.In the late 1970s, anomalies in the behaviorist model began to appear with increasing frequency (Ainsworth, 1979) and perhaps the most damaging criticism was the charge that the approach favored lower-order thinking skills at the expense of higher-order outcomes (Reboy & Semb, 1991), which resulted in the shift in thinking to cognitive models.
The criticisms of behaviorist approaches were coupled with related research inquiry into the efficacy of multiple-choice tests in assessing higher-order thinking skills (Crook, 1988;Haladyna, Downing, & Rodriguez, 2002).Practitioners found a speedier resolution to this latter challenge.Despite even their inventor, Frederick Kelly, describing multiple-choice tests as "lower order thinking of the lower orders" (as quoted in Godin, 2012), it became increasingly apparent that the format could be adapted to higher-order thinking skills with a certain degree of coaching (Haladyna & Downing, 1989;McCurry, 2008;Morrison & Free, 2001;Owen & Freeman, 1987).A variety of handbooks, guides, and related information cropped up and such testing strategies became part of the standard curriculum for instructors (Brookhart, 2010;Kominski, 2012;McKeachie, 1994;Miller, Reed, & Haladyna, 1978).The question of higher-order thinking and objective testing remains contested (Svenningsen & Pear, 2011), but it would appear that if there are deficiencies, they are perhaps not as dire as predicted.Capitalizing on best practices, the instructor in this study designed the multiple-choice questions carefully to span a range of cognitive outcomes, from lower-order understanding to higher-order problem solving.For example, one lower-order question asked students to identify the typical royalty rate for licensed merchandise, while a higher-order problem called on students to discern the strongest justification for increasing ticket prices to games.
Subsequent analysis looked more closely at the data gathered from the course, comparing results on lower-order and higher-order thinking questions.This article's second author, who had no input on the course design, curriculum, or assessment, coded each multiple choice question based on Bloom's Taxonomy (Bloom, 1956).Questions at Bloom levels 1 and 2 were considered lower-order thinking questions; those at Bloom levels 3 and 4 were higher order (samples of questions are included in the Appendix).There were no questions at Bloom levels 5 or 6 (King, Goodson, & Robani 1998). 6hen looking at the original dataset (N=114), Shapiro-Wilk tests for normality in the differences indicated that normality was violated among higher-order questions (S-W=0.970,p=0.012), and an analysis of extreme values showed two outlier cases where improvements were well beyond the norm.After excluding these cases, the differences in the higher-order dataset were normally distributed (N=112; S-W=0.980,p=0.091).Differences for lower-order questions were normally distributed with the full dataset (N=114; S-W=0.990,p=0.547) and the dataset with the two [higher-order] outliers removed (N=112; S-W=0.984, p=0.198).However, a box plot on the lower-order differences revealed a new outlier in this data, which was then removed.In order to have a consistent sample, all three outliers were removed from all analyses, including those mentioned above.
Using the new dataset (N=111), the differences between the quiz scores and final exam scores are normally distributed for both lower-order (S-W=0.985,p=0.244) and higher-order (S- There was strong support for improvement in questions assessing lower-order thinking (p<0.05), which is consistent with a priori assumptions.Students improved 1.45% between the quizzes and the final exam for Bloom level 1 and 2 questions.In a very surprising result, the improvement on higher-order questions was even greater (4.05%) and was statistically significant (p<0.0001).Considering instructors' past struggles with using competency-based education for higher-order questions, these results suggest that hybrid models, blending best practices from both behaviorist and cognitive approaches, may have the potential to dampen some of the previously identified negative effects and to positively affect student learning outcomes.
Much of the findings of this study were contrary to the existing literature and instructor's intuition, so this study reinforces the value of data-based analysis in the scholarship of teaching and learning.The primary finding of the study, though, was the affirmation that the hybrid course design described here can be a potentially effective means to develop students' higher-order thinking.For these reasons, the instructor for this course has already taken efforts to improve his writing of multiple choice questions that target higher-order thinking and other related interventions.This preliminary study suggests that reviving research into the design and assessment of competency-based models within the contemporary context of higher education may not be an exercise in beating a dead horse.Rather, thanks to gains in other areas of pedagogical research, the questions that arise from this study could contribute to a productive line of new inquiry.
One question the study does not answer is student perceptions of the hybrid model.Whether or not this kind of learning is meaningful to them remains an open question.Other studies of student and stakeholder perceptions of competency-based approaches have largely reported favorable (though not unanimously so) results (Bell & Mitchell, 2000;Dhillon & Moreland, 1996;Reynolds & Sharpe, 1992), an outcome that resonates with feedback from student evaluations of the course under study.That being said, studies that use student perceptions of instruction have come under increasing scrutiny and have not held up well as indicators of learning, particularly in the case of innovative pedagogies (Bowman, 2010;Gonyea, 2005;Mabe & West, 1982).This may be especially the case in neo-behavioral models, where student perspectives are not an integral part of the learning theory.That is not to say that qualitative studies would not be valuable, rather to say that the design of such a study would need to take into account the changing pedagogical context.
Further, such inquiries would have to be conducted with few examples to guide them.This reflects another significant caveat to interpreting this study's results-research in contemporary competency-based pedagogy suffers from a shortage of meaningful benchmarks.While these results show that students may have made modest progress towards their learning outcomes, or in mastering these competencies, it is difficult to say whether or not students would have been equally, or perhaps even more, successful with the previous instructional framework or in the hands of a different instructor.This study, based on a single course taught by a single instructor at a single institution, did not have a control group nor were there many other courses or studies to which this kind of data could be compared.In other words, this study is very limited in both scope and scale, so the picture it provides is a small one.
Part of this benchmarking challenge is an expected product of a paradigm shift.When Thomas Kuhn first popularized the term in the 1960s, he argued that a paradigm shift would result in exactly this kind of incommensurability, in which the world would appear as if through different lens or as if spoken in a foreign language after a paradigm shift had occurred (Kuhn, 1970).If neobehaviorism does constitute such a paradigmatic shift in higher education, then a similar incommensurability may affect educational research.In other words, the new question of "are you competent" cannot be meaningfully answered until (or if) others choose to tackle the assumptions and practices associated with neo-behaviorism and make the results of their work available as benchmarks (Twigg, 2005).
This kind of robust benchmarking may or may not occur.As mentioned previously, competency-based instruction challenges many of the assumptions that underlie contemporary university practice-assumptions that many are prepared to defend.The approach has been the center of vociferous criticism, both in the popular and academic press (Chappell, 1996;Chappell, C., Gonczi, A. & Hager, 2000;Hodkison & Issit, 1995;Mendenhall, 2012;Pennington, 1994;Slaton, 2013).While these critiques take multiple tacks, faculty concerns often focus on the perception their diminished role in the teaching process and the implications of that shift for the quality and rigor of student learning.This study does not directly address this issue, but it does suggest that competency-based approaches do not have to envelop or overtake others, but rather can be used to supplement existing pedagogical tool kits, providing another possible means to vary teaching to meet the evolving needs, both cognitive and behavioral, of students.In such cases, the selection, application, and development of appropriate and effective instructional approaches remain in the domain of the faculty member, whose expertise in both pedagogy and the subject matter are necessary to determine (and assess) the right blend.Whether or not the prognosticators of the neo-behavioral paradigm shift prove victorious, this study suggests that there may be value in co-opting these methods into an intentionally balanced teaching portfolio.
Contemporary discourse is replete with questions about the purpose of higher education, the nature of the undergraduate experience, and the (real or perceived) value of a college degree (Bowden & Masters, 1993;Wieder, 2011).All of us in higher education face choices with these kinds of disruptive challenges, and our response does not have to be defensive.As the Indian philosopher M. J. Handa (1986) admonished, educational institutions do have the ability to extricate themselves from their conventional "paradigm paralysis."Instead, they can embrace new ways of thinking about what and how students learn, including perhaps a re-examination of the neo-behavioralist assumptions that underlie competency-based pedagogy.The experience of this study, in its own small and limited way, underscores the value of active experimentation as a catalyst for intentional responses to potential pedagogical paradigm shifts.
Tyler and Cruz Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 3, June 2016.josotl.indiana.edu26 Tyler and Cruz Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 3, June 2016.josotl.indiana.edu27

Table 4 : Results of paired t-tests comparing quizzes to final exams based on question type
Tyler and Cruz Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 3, June 2016.josotl.indiana.edu30 W=0.979, p=0.079) questions, and box plots and Q-Q plots confirmed that there were no outliers.Given that the requisite assumptions are met, researchers conducted paired t-tests to compare the means (see Table 4). 7