A Mixed-methods Evaluation of Teaching Evaluation: Innovative Course-Based Service-Learning Model on Program Evaluation Competencies

: There is limited empirical study that examines effective evaluation pedagogy on evaluation competency outcomes. The purposes of this article are to (1) identify key course design features critical to integrating real-world projects in a course-based service-learning model; and (2) provide empirical data using mixed-methods evaluation to assess the impact on evaluation competencies. Data from 5 cohort doctoral students (2015-2019; n=51) showed such course design resulted in significantly increased before-after course scores on the 6-item course learning objectives (scale item means of 3.34 vs. 4.65; p<.001), and the 17-item program evaluation competency scale (PECS-17) (scale item means of 3.25 vs. 4.76; p<.001). Qualitative reflections on the five essential evaluation competency domains convergently were correlated with strong positive competency outcomes. The convergent findings from both quantitative and qualitative data provide strong empirical evidence of evaluation competencies gained. These findings have implications on teaching evaluation of graduate students for evaluator educators who strive to provide competency-based experiential learning.


Hou
process. Poth and colleagues applied a systems approach to teaching evaluation. They used a mixedmethods case study examining four of the five essential evaluation domains (minus the PMP domain) among a small group of graduate students to understand intended and un-intended learning competencies (Poth et al., 2020). Linfield (2019) compared a real-world experiential evaluation program versus hypothetical evaluation projects, and results showed students gained limited value from the theoretical evaluation training. More engaging content with real-world practice and application, combined with team-based learning, has been recommended to improve students' learning outcomes while also benefitting community partners (LaVelle, 2020;Linfield, 2019;Birkby & Linfield, 2019;Bakken, Nunez, & Couture, 2014). Empirical studies have shown that the coursebased service-learning approach results in powerful impacts and significantly increased student confidence in program planning competencies (Hou 2009) and program development and implementation competencies (Hou and Pereira, 2017).
Limited teaching research has examined evaluation competencies as learning outcomes. Ridde and colleagues were among the first to include evaluation competency assessment as a summative evaluation among master students taking an evaluation course in Africa's population health program (Riddle, Fournier, Banza, Tourigny, & Ouedraogo, 2009). Dillman (2012) examined evaluator competencies (i.e., contextual, management, communication, and methodological, theoretical knowledge), educational experience among novice evaluators, and their relationship. Findings indicate fieldwork contributed to the development of evaluation competencies more so than any other educational experience. Mentoring from the instructor played the second most crucial supporting role to evaluator competency development (Dillman, 2012). Still, there remains a need to have continued evaluation-specific guidance in designing and implementing an effective competency-based evaluation regime for higher education instructors and competency-based measurement for assessing the complex learning impact (Poth et al., 2020).

Purpose
To better equip our future evaluators in assessing these essential evaluation competency domains identified, creative teaching approaches incorporating hands-on practice to real-life evaluation are critical to engaging learning (LaVelle, 2020;Linfield, 2019;Hou and Pereira, 2017;Dillman, 2012). Yet, limited empirical studies have examined the role opportunities, such as integrating real-world community experience with the hands-on practice via course-based service-learning projects, contribute to evaluation training within higher education settings. Existing limited studies also use varied competency items (Riddle et al., 2009;Dillman, 2012;Poth et al., 2020). Few discussed the utility of mixed-methods evaluation on evaluation program competencies learning outcomes (Poth et al., 2020). The purposes of this article are two-fold: (1) to identify key course design features critical to integrating real-world projects via a course-based service-learning model for teaching evaluation, and (2) to provide empirical data of using mixed-methods evaluation to assess the impact of such innovative course design on program evaluation competencies. Findings from the current empirical data will provide evaluator educators practical guidance on key design features to develop and implement competency-based experiential education and an effective brief competency-based mixedmethods tool to assess student learning impact before and after the evaluation.

Design of the Policy and Program Evaluation Course
Under an intentionally structured and designed course learning environment, early exposure to realworld community partners, coupled with faculty supervision and guidance, are critical before students enter internship or career phases. The Policy and Program Evaluation is a core course for an interdisciplinary Doctoral Program in Public Affairs at a large southern university in the USA. The Policy and Program Evaluation course's key innovation is the infusion of course-based servicelearning projects combined with interdisciplinary team-based learning in cooperation with real-world community partners. The purpose of various course-based service-learning projects with area agencies and organizations was to infuse hands-on, real-world learning and application experience. The instructors assigned interdisciplinary teams of students with consideration to students' research interests and prior experience. The primary purposes of integrating these course-based servicelearning projects are to provide real-world, hands-on practice learning opportunities to build evaluation competency across all essential domains (Linfield, 2019;Dillman, 2012).
A utilization-focused evaluation approach was used, inviting project managers and key staff partners to participate in the evaluation design and evaluation proposal presentation to provide feedback (Ramirez, R., Brodhead, D., & Quarry, W., 2018). In this course, students were assigned to small interdisciplinary teams to work with community stakeholders developing authentic and workable evaluation proposals. This course helped facilitate a sense of ownership over processes and findings and promoted evaluative thinking. Additionally, building a campus-community partnership promoted engaged student learning and provided technical help to evaluate programs addressing complicated social issues (Bakken, Nunez, & Couture, 2014).
Guided reflection opportunities were integrated early in the semester before the first stakeholder meeting, throughout the semester, project discussions, and overall personal reflection at the end of the course (Conway, Amel, & Gerwien, 2009). Such innovative teaching approaches aimed to engage students in high-level critical thinking while working with real community partners to develop practical evaluation plans for assessing the impact of essential programs to address complex social and health issues among underserved groups. Such a design sought to strengthen student evaluation competency development in situational practice, real-world planning, management practice, interpersonal communication domains, professional practice, and methodology technical practice domains.
Instructors' reflections, qualitative course feedback, and end-of-course reflections from five cohort groups of doctoral students (n=51) were analyzed using thematic analyses to identify lessons learned on key course design themes focused on ensuring high-quality evaluation competency development (Creswell, 2016).

Mixed-methods Assessment on Student Program Evaluation Competency Outcomes
A mixed-methods approach was used to systematically examine and analyze evaluation competencies among students and the five essential domains of evaluation competencies. The power of mixedmethods evaluation lies in the added value of integrating both quantitative and qualitative data, providing nuanced insights that cannot be gained when only a single type of data is measured. Mixedmethods research design brings together quantitative numbers' strengths with rich, contextual, qualitative information to examine student learning more holistically (Creswell & Plano Clark, 2018). Essential evaluation competencies were assessed with quantitative and qualitative questions to allow corresponding comparisons on key variables (Creswell & Plano Clark, 2018;Hou, 2020 Hou from both the quantitative and qualitative strands were analyzed separately, then convergently drawn upon to interpret and conclude student learning outcomes. Quantitative Measures. In addition to a 6-item course learning objective scale (Table 1), a 17-item Program Evaluation Competency Scale (PECS-17) measuring detailed performance objectives for developing a sound and practical evaluation proposal was developed and tested. Quantitative measurements were developed via (1) review of content coverage and skills recommended by key evaluation textbooks (McDavid, Huse, Hawthorn, 2019;Newcomer, Hatry, & Wholey, 2015;Mertens & Wilson, 2012;Centers for Disease Control and Prevention, 2011;Rossi, Lipsey, & Freeman, 2003), (2) examination of course learning objectives and topic outlines, and (3) correlation to essential evaluation competency domains recommended by AEA and CES. Key competencies identified were translated into corresponding quantitative performance objectives measurement items, and then pilot tested among a small group of graduate students. These step-by-step performance objectives were carefully built in sequentially, practiced, and discussed during weekly class meetings. Also, two items were developed to assess the overall course impact on student evaluation competencies at the end. Table 2 provides detailed item descriptions with statistics. Paired t-tests were used to examine program evaluation competencies before and after the course.
Qualitative Measures. Five qualitative reflective questions were developed to examine the five essential domains key to evaluation competencies and gain a more holistic perspective of course impact on student learning. Students were asked to reflect, at the end of the evaluation course, on each of the essential evaluation competency domains, and describe (with scenarios or examples if possible) how well or competent students felt they were, as a result of this evaluation course and project, as compared to the beginning of the semester (Table 3). Open-ended qualitative end-ofcourse feedback was also analyzed to identify key course design features instrumental to students' competency development.
Data Analyses. Students enrolled in the doctoral program evaluation course during 2015-2019 completed the 6-item course learning objectives and 17-item program evaluation scale survey at the beginning and end of the course (n=51). Descriptive statistics, item-total correlation, and Cronbach's alpha coefficients were calculated to evaluate the internal consistencies. Data from before and after courses were used to compare changes in program evaluation competencies via paired t-tests (Hou, 2020).
These five cohort student groups also provided qualitative feedback on their course experience and learning impact (n=51). In addition, students from the 2019 class (n=8) also completed the five qualitative questions to reflect on their learning as related to the five essential domains of evaluation competencies specifically. An inductive approach was used to analyze these qualitative data. The instructor first read through all the database comments to gain a general familiarity, then conducted a line-by-line coding approach, assigning a code label to the text segments. Next, these codes were aggregated into themes for the current study's qualitative report (Creswell, 2016). Thematic analysis with sample quotes provided was used to identify both the impact on essential competency domains (Table 3) as well as key course design features (Table 4). Quantitative and qualitative data were crossexamined and compared for integrated interpretations of the study findings' convergences and divergences. The human subject office approved the study of PI's institution (IRB# STUDY00001671).
(1) Overall, a carefully structured course with sequential design to facilitate progressive learning. Students enjoyed the structure and sequence design of the course. Students commented how motivating it was that "each week we learned something that could be immediately used in our final proposal" and "class discussions were lively and useful." Also, "[t]he sequence of course content, alignment of activities with the course objectives, and the overall organization… helped… remember the essentials…" (2) Incremental and relevant assignments for continuous feedback. To ensure sufficient progress and quality development of the evaluation proposal, multiple intermediate progress reports were built in for students to get input from stakeholders and receive feedback from both the instructor and student peers. Students commented, "the partial submission of each component . . . really assisted with refining the final product as we could get peer and expert [the instructor] feedback." (3) Integrated hands-on, real-world service-learning experience to deepen engagement. Students appreciated the engagement with stakeholder activity, which was crucial to deepening students' understanding of communities' authentic life issues. Gaining the "critical insights on addressing potential challenges and issues during the evaluation process" was invaluable. "The ability to connect directly with stakeholders" has been "an amazing opportunity" which students really appreciated, and "learning content and applying it to the project allowed… [us] immediately [to] use what [we] learned in realworld situations." (4) Guided personal reflections to link academic learning with project experience. Students practiced guided reflection early in class before interacting with community stakeholders. In-class group debriefing was facilitated to address anxiety, concerns, or misconceptions students may have had. Students reflected they now "know what questions to ask, [and] what direction to take to construct a strong and thorough evaluation proposal." (5) Constructive, safe, and relaxed learning environment boosted learning. Studies indicate emotion plays a significant role in affecting humans' cognitive process, including learning and memory (Tyng, Amin, Saad, and Malik, 2017). Creating a caring and safe environment is critical to help ease anxiety and facilitate learning. Student feedback reinforced this critical yet often overlooked factor: "[the instructor] created a learning environment that is encouraging and conducive to classroom discussion surrounding the topic." " The instructor made this class fun and interesting . . . facilitated a meaningful course," and "demonstrated interest in helping me succeed, . . . boosted my confidence and enabled me to relax and learn!" Finally, Table 5 showcased weekly course topics with readings and assignment activities and corresponding competency-based measurement items to provide practical guidance on the course design and logistic timeline for teaching implementation. Article reviews (AR) were arranged throughout the semesters. Each article review engaged a pair of students from different disciplines to analyze and critique an evaluation article of students' choice to encourage critical thinking. The instructor taught essential evaluation concepts through the weekly informative lectures with examples and exercises to build the week's targeted evaluation competencies. Partial submissions of each evaluation proposal component with feedback from peers and instructors were built in throughout the semester, including an early progress report, regular checkpoints with stakeholders, project discussions, peer critics on drafts, and final oral and written proposals. These were aimed to challenge students to balance the rigor required for evaluation with stakeholder perspectives and constraints in real life while assisting with refinement for quality final proposal development. Finally, miniassignments were designed to engage students with a deeper understanding and application of the various threats to validities and different evaluation designs to further strengthen the critical evaluation competency in the methodology technical practice domain. Further course design elements and pedagogy details were documented elsewhere (Hou, in press).

Discussion
Integrating course-based interdisciplinary service-learning projects to bridge stakeholder engagement and authentic learning demonstrated a profound impact on developing (training) program evaluation competencies among students. Quantitative data showed such engaged community service-learning experience with hands-on practice in real-life evaluation projects significantly increased confidence in the 6-item course learning objectives, the Program Evaluation Competency Scale (PECS-17), and the two overall evaluation competency assessment. Also, qualitative reflections on the five essential domains of evaluation competencies convergently showed positive outcomes. These convergent findings from both quantitative scales and qualitative reflections demonstrated strong empirical evidence of program evaluation competencies gained among study participants.
Identifying meaningful community service opportunities linked to carefully designed course activities addressing academic learning objectives requires experience and training (Hou and Wilder, 2015;Hou, 2010;Hou, 2009). Such a high-impact learning approach is time-consuming and needs significant advanced planning and follow-up debriefing to continue building positive campuscommunity partnerships. Faculty teaching and learning workshops on course-based service-learning pedagogy and strategies are recommended to better prepare competent instructors for developing qualified future evaluators.
Bridging the gap between academic learning and community implementation can reward all parties involved, including students, community partners, faculty, and the institution (Hou and Wilder, 2015;Hou, 2010). Creating evaluation proposals for community partners who may adopt and implement has been a life-changing experience that many students cherish. The current empirical study shows "[i]nteracting first-hand directly with real community partners [has] been very eyeopening" and re-shaped students' views towards high-risk communities, besides developing "increased self-confidence we didn't have before." Community partners have genuinely appreciated the "professional and objective insight students provided" to their projects and assessments. Such an arrangement helps partners "reflect on their own practice from new points of view." The first-hand experience, seeing how such course-based service-learning approaches can deepen and internalize skills learned, can reinforce an instructor's passion and commitment to such teaching and learning approaches. This is also "an important way to fulfill the civic roles of higher education institutions and increase university's visibility while building trusting academic-community partnerships" (Bakken et al., 2014;Conway, 2009;Cauley et al., 2001).
A few limitations of this study should be noted. This mixed-methods evaluation was a onearm before and after study with no comparison group and subject to potential validity threats (Rossi et al., 2003). Although data were collected during 2015-2019 from five cohort groups of students, the sample size was still relatively modest. In addition, only the 2019 cohort group was given the five qualitative questions specifically probing the five essential evaluation domains. The PECS-17 was also not developed with the purpose to equally map out the five essential competency domains. Instead, key step-by-step performance objectives critical to designing a complex evaluation study were identified to address more of the evaluation competencies' technical skills domain. Thus, not surprisingly, nearly 60% of the competency items measured were primarily in the methodology technical practice domain. The study used the five-item qualitative measurement tool to examine student learning in all five essential competency domains in a more balanced way. Additional items may be added to allow for potential sub-scale analyses in the future. A larger sample size and comparison group would also strengthen this mixed-methods evaluation design. All cohorts were taught by the same instructor, and the course's success could be due to the instructor's skills in addition to the course's design.

Implications on Teaching
This study contributes to the much-needed empirical study data examining the role of course-based service-learning opportunities that integrate real-world experience with hands-on practice for student evaluation competency development. The current study's empirical data demonstrates such teaching approaches have significantly impacted student development and showed significant increases in evaluation competencies and student confidence. This study identified five key course design features with empirical student quotes illustrating the impact on student learning. The PECS-17, along with the five qualitative questions examining essential evaluation competency domains, provide a short yet effective research-tested mixed-methods measurement tool to assess evaluation competency outcomes. This study has implications on teaching evaluation among graduate students. Evaluator educators who strive to provide competency-based experiential learning to develop competent evaluators can gain practical guidance on key course design features and corresponding competencybased measurement tools to assess evaluation competency outcomes.  [The instructor] keeps everyone on a really reasonable timeline that ensures the completion of our papers in a timely manner while reaching our learning objectives too.

Appendix
Integrated hands-on real-world servicelearning experience to deepen engagement The course has challenged me to assess evaluation outcomes from different stakeholder perspectives. I gained invaluable critical insights on addressing potential challenges and issues during the evaluation process.
The ability to actually create evaluation proposals for programs that may actually be implemented is an amazing opportunity which I appreciated.