The Road to Redemption: Reclaiming the Value in Assessment Retention Exams

: A good assessment plan combines many direct and indirect measures to validate the collected data. One often controversial assessment measure comes in the form of retention exams. Although assessment retention exams may come with faults, others advocate for their inclusion in program assessment. Objective-based tests may offer insight to instructors about course objectives that students comprehend well and other concepts that need more attention. This research shows that using retention exams as an assessment measure can generate useful and meaningful data for both the students and the program. Students can learn strengths and weaknesses based on scores. Faculty and programs can learn where gaps may exist within the program. But, whenever a program decides to use retention exams as an assessment tool, faculty members need to be included in the process. Discussions about content need to occur constantly. Exams need to consistently reflect current standards and student learning objectives. And faculty need to stay involved in the process to know exactly where any inconsistencies may lie within their courses, and how they contribute to the students’ overall experience within the program.

has been 5 or 6 years since the student has been exposed to the material, then review sessions might be required (Tucker, 2006, p. 379-80).
Although assessment retention exams may come with faults, others advocate for their inclusion in program assessment.Banta and Palomba (2015) argued that "tests that contain wellwritten items covering the appropriate subject matter and level of thinking can reveal much about student learning" (p.110).Objective-based tests may offer insight to instructors about course objectives that students comprehend well and other concepts that need more attention.Tucker (2006) agreed and said to make the exams become part of a regularly graded class such as a capstone experience.Students expect these types of exams to be part of a course because they expect to be tested on course material.Since the capstone class encompasses the entire program, students expect some sort of testing of their knowledge and skills of the program prior to their graduation.In the case of the broadcasting program at the university used as the subject for this paper, the exams became part of the capstone class, and featured appropriate subject matter to reveal true evidence of student learning.However, the road to recovery for this assessment measure hit a few bumps along the way.This paper shares the importance and the value of a systematic approach to assessment regarding the data gathered, and also provides a way to create commitment within the entire department or program to strengthen their inclusion as an effective assessment tool.

Initial Analysis of Retention Exams
When first presented with the capstone course and its contents, the instructor, as any new instructor to a course might do, utilized the resources from the previous instructor.Although the materials were comprehensive, the assessment exam portion of the class lacked efficiency in the execution of the exams and consistency of the construction of the exams as well.The first issue the instructor addressed was the efficiency in the distribution and completion of the exams.The instructor utilized the learning management system in use by the university to construct electronic examinations so tests could be immediately scored, and so data from the tests could be collected and aggregated more efficiently.This only scratched the surface of the issue.The collected data still reported inaccurate information as the tests were not normed or consistent.Tucker (2006) found that "given the lack of nationally normed exams in the area of mass communication, any department that decides to use an exit exam as a measure of direct assessment will have to create one" (p.378).The broadcast department in this study constructed the exams, but lacked the appropriate guidance to make them able to collect rich, useful assessment data.The second step to redemption for the retention exams as an assessment instrument needed to tackle this issue.

A New Paradigm
The key to creating useful retention exams for assessment starts with involving the faculty in their creation.Banta and Palomba (2015) agreed that "tests that are developed locally for program assessment typically reflect a group effort" (p.108).Suskie (2009) also pointed out that "our assignments are more effective if we first clarify what we want students to learn from the assignment and then design an assignment that will help them achieve those ends" (p.148) and objective tests "are especially good for assessing fundamental knowledge and understanding" (p.166).As a result, the instructor charged the faculty with creating retention exams that reflected fundamental knowledge and understanding, and emphasized retention of course and program objectives.
The retention exam creation started with a test blueprint.A good objective test that uses a test blueprint ensures that tests reflect course objectives and learning goals deemed important by the department (Banta & Palomba, 2015;Suskie, 2009).The department simplified this process by using the Master Course Syllabus (MCS) required for every course as the template.The MCS, required by the university, outlines each course by providing a description, Student Learning Outcomes/Objectives (SLO), and suggested course evaluation measures.The student learning outcomes and course objectives listed on the MCS clearly indicate the desired outcomes of the course.Each question on the exam needed to reflect these measures and not any text-or instructorspecific content.Although the MCS simplified the process, the test creators still needed some additional guidance.Tucker ( 2006 The broadcast department faculty created the new retention examinations under the guidance of the assessment coordinator and course instructor.Each instructor reviewed the existing version of the exam to compare questions on the exam against student learning outcomes on the MCS.Any question that did not reflect one of the course objectives was eliminated and replaced with one that did.For example, one SLO required students to "identify a variety of methods of mass media criticism and analysis including aesthetic, sociological, economic, structural, psychoanalytical and ethical perspectives."The original multiple choice question asked students to define "the Way of the Rhetorician."This was a question specific to the textbook, and not covered by all instructors of the course.A new question addressing this SLO asked students, in multiple choice format, to define Auteur Theory.This new question addressed a basic tenet of the course, covered by all instructors of the course.Each instructor created their objective-based exams in this manner with multiple choice, true/false, and matching-style questions using the MCS objectives as a blueprint.
The following semester the capstone course instructor deployed the new exams through the learning management system as before.Although the new tests generated improvements in scores, new concerns arose from the faculty, and new problems occurred in the classroom.
The faculty questioned, now that the exams were taken electronically though the learning management system, if the capstone course was the appropriate place to administer the exams.Students previously enrolled in the course were not held accountable for their performance on the exams, so the faculty questioned the authenticity of the data and the purpose of their inclusion in the course.The instructor and assessment coordinator argued for the continued inclusion of the exams as part of the capstone class for the program.Suskie (2009) supported this argument, stating "capstones help students synthesize their learning by tying together the various elements of their program and seeing the big picture" (p. 7).The assessment retention exams administered in the course reflected this synthesis.Additionally, Banta and Palomba (2015) agreed that "objective tests are a normal and expected part of the classroom experience and are a type of direct measure included in many assessment programs" and "objective tests allow students to demonstrate the knowledge they have acquired and their ability to process and use that knowledge" (p.105).So the exams stayed.But the question then became not about whether the exams should stay part of the capstone experience, but how to hold the students accountable for their performance.

Addressing Accountability
The assessment coordinator for the program suggested including the exam scores as part of the course grade.Banta and Palomba (2015) supported this argument noting "with locally developed assessment instruments, faculty are often more comfortable with test content and its relationship to the curriculum, and are more likely to include results in the course grade" (p.109).Faculty questioned the fairness of including scores on test material covered in courses anywhere from two to five years prior to when the students take the exams.Informing the students prior to taking the exams of their value, and the inclusion in their course grade effectively tackles this issue.Banta and Palomba (2015) addressed this concern as well, noting that "because of the many possible approaches, students must be informed of the effect, if any, that their performance or participation will have on their grades" (p.109).The course instructor suggested holding the students accountable for only a portion of the scores on the exams.This way the exams would encourage the students to perform at their highest level without penalizing them too severely for poor retention.The faculty seemed amenable to the compromise.But, this required more changes to the exams.
Each exam needed to contain the same number of questions to simplify the point process.Students enrolled in the capstone class completed a total of eight different assessment exams, one for each core class in the curriculum and one elective exam reflecting their chosen concentration within the program.The proposed method for inclusion scored all eight exams, but only held the students accountable for their performance on seven.The instructor aggregated the scores on the exams and divided by one less.Students who performed well could potentially score higher than 100%, but students who failed to perform well would not suffer harshly for their performance.With all parties in agreement and new tests to deploy, the new semester looked promising for the assessment coordinator and capstone course instructor.

Data Collection and Analysis
Data from the assessment exams in the semester following deployment of the new procedure provided encouraging results.Students scored higher than in previous semesters thanks to uniform exams that reflected course objectives.Banta and Palomba (2015) mentioned that incentives for students to participate in assessment activities may be necessary, but noted that intrinsic motivation always elicits their best work.Tying the score on the exams to their course grade worked as the motivating factor in this case.Additionally, the instructor communicated the importance of the exams to the students, noting that the data collected help to make the courses in the program stronger for future students.assessment, students are much more likely to care too" (Banta & Palomba, 2015, p. 62).The inclusion of the scores in the final course grade, and the discussion of the implications of the results, encouraged students to perform at their best level.The electronic distribution of the exams allowed the department to collect data for instructors to evaluate which concepts needed more attention, and those concepts students retained more successfully.The instructors used this data to ensure instruction of course concepts remained even across all instructors, and that those concepts where students were lacking in retention were reinforced.Additionally, instructors could alter exams if questions were continually missed.Although the process still needed some adjusting, the introductory phase of deployment encouraged the course instructor and assessment coordinator to continue their efforts.

Conclusions
The use of assessment retention exams is back on the rise.Banta and Palomba (2015) noted that "the percentage of program heads reporting that they use this assessment method is above 50 percent for trade programs, health sciences, computer science, business, and engineering" (p.106).In the broadcasting program at the university in this paper, assessment efforts continue to look for ways to use multiple assessment instruments to gather useful, meaningful, and rich data.Suskie (2009) reminded programs and assessment coordinators to "make participation in the assessment a requirement of a program or course (typically a capstone course)" (p.29).Banta and Palomba (2015) prompted test designers to ensure that tests are not focused at "such a general level of information that they do not yield detailed results useful for improvement of teaching and learning" (p.107).
This research shows that using retention exams as an assessment measure can generate useful and meaningful data for both the students and the program.Students can learn strengths and weaknesses based on scores.Faculty and programs can learn where gaps may exist within the program.But, whenever a program decides to use retention exams as an assessment tool, faculty members need to be included in the process.Discussions about content need to occur constantly.Exams need to consistently reflect current standards and student learning objectives.And faculty need to stay involved in the process to know exactly where any inconsistencies may lie within their courses, and how they contribute to the students' overall experience within the program.
The data continue to flow, and the results continue to generate useful information for program evaluation.But the process needs to continue to constantly be reevaluated.Without continuous monitoring and attention to exams to ensure their reflection of changing course objectives, the critics of retention exams will continue to devalue retention exams as useful assessment measures.For this program, the road to redemption lead to richer more useful data than any collected before.
) suggested four helpful techniques: 1.The content of the tests should match the classroom objectives and what the teacher emphasized.2. The test items should represent the full range of knowledge and skills that are the primary targets of instruction.3. Expectations for student performance should be clear.4. The assessment should be free of extraneous factors, which unnecessarily confuse or inadvertently cue student responses.(p.375)

Figure
Figure 1: Development Cycle

Table 1 . Comparison results from old instrument to new instrument
"The messages faculty give about assessment are powerful motivators.If faculty care about StohlmanJournal of the Scholarship of Teaching andLearning, Vol. 15, No. 5, October, 2015.Josotl.Indiana.edu69