A Mixed Methods Evaluation of Teaching Evaluation: Innovative Course-Based Service-Learning Model on Program Evaluation Competencies
Article Sidebar
Published:
Mar 31, 2022
Keywords:
course-based service-learning, program evaluation competency, mixed methods evaluation, teaching evaluation, competency assessment
Main Article Content
Abstract
Abstract: There is limited empirical study examines effective evaluation pedagogy on evaluation competency outcomes. The purposes of this article are to: (1) identify key course design features critical to integrating real-world projects in a course-based service-learning model; and (2) provide empirical data using mixed methods evaluation to assess the impact on evaluation competencies. Data from 5 cohort doctoral students (2015-2019; n=51) showed such course design resulted in significantly increased before-after course scores on the 6-item course learning objectives (scale item means of 3.34 vs 4.65; p<.001), and the 17-item program evaluation competency scale (PECS-17) (scale item means of 3.25 vs 4.76; p<.001). Qualitative reflections on the 5 essential evaluation competency domains convergently were correlated with strong positive competency outcomes. The convergent findings from both quantitative and qualitative data provide strong empirical evidence of evaluation competencies gained. These findings have implications on teaching evaluation of graduate students for evaluator educators who strive to provide competency-based experiential learning.
Downloads
Download data is not yet available.
Article Details
How to Cite
Hou, S.-I. (2022). A Mixed Methods Evaluation of Teaching Evaluation: Innovative Course-Based Service-Learning Model on Program Evaluation Competencies. Journal of the Scholarship of Teaching and Learning, 22(1). https://doi.org/10.14434/josotl.v22i1.31705
Section
Articles
This work is licensed under a Creative Commons Attribution 4.0 International License.
- Authors retain copyright and grant the Journal of the Scholarship of Teaching and Learning (JoSoTL) right of first publication with the work simultaneously licensed under a Creative Commons Attribution License, (CC-BY) 4.0 International, allowing others to share the work with proper acknowledgement and citation of the work's authorship and initial publication in the Journal of the Scholarship of Teaching and Learning.
- Authors are able to enter separate, additional contractual agreements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in the Journal of the Scholarship of Teaching and Learning.
- In pursuit of manuscripts of the highest quality, multiple opportunities for mentoring, and greater reach and citation of JoSoTL publications, JoSoTL encourages authors to share their drafts to seek feedback from relevant communities unless the manuscript is already under review or in the publication queue after being accepted. In other words, to be eligible for publication in JoSoTL, manuscripts should not be shared publicly (e.g., online), while under review (after being initially submitted, or after being revised and resubmitted for reconsideration), or upon notice of acceptance and before publication. Once published, authors are strongly encouraged to share the published version widely, with an acknowledgement of its initial publication in the Journal of the Scholarship of Teaching and Learning.
References
References
AEA Evaluator Competencies (Revised April 5, 2018). American Evaluation Association. Retrieved Oct. 21, 2020, from https://www.eval.org/page/competencies.
Association of Experiential Education (n.d.). What is Experiential Education? Retrieved Nov. 10th, 2020, from http://www.aee.org/what-is-ee.
Bakken, L. L., Núñez, J., & Couture, C. (2014). A course model for building evaluation capacity through a University–community partnership. American Journal of Evaluation, 35(4), 579-593.
Centers for Disease Control and Prevention. (2011). Office of the Director, Office of Strategy and Innovation. U.S. Department of Health and Human Services. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention.
Cauley, K., Canfield, A., Clasen, C., Dobbins, J., Hemphill, S., Jaballas, E., et al. (2001). Service learning: Integrating student learning and community service. Education for Health, 14(2), 173-181.
Conway, J.M., Amel, E.L., & Gerwien, D.P. Teaching and learning in the social context: A meta-analysis of service learning’s effects on academic, personal, social, and citizenship outcomes. (2009). Teaching of Psychology, 36: 233-245. https://doi.org/10.1080/00986280903172969
Christie, C. A., Quiñones, P., & Fierro, L. (2014). Informing the discussion on evaluator training: A look at evaluators’ course taking and professional practice. American Journal of Evaluation, 32 (2), 274–290.
Creswell J. (2016). 30 essential skills for the qualitative researcher. Sage Publications, Inc.
Creswell, J., & Clark V.L. (2018). Designing and conducting mixed methods research. (3rd ed.) Sage publications, Inc. ISBN: 978-1-4833-4437-9
Darabi, A. (2002). Teaching program evaluation: Using a systems approach. American Journal of Evaluation, 23(2), 219-228.
Davis, M. V. (2006). Teaching practical public health evaluation methods. American Journal of Evaluation, 27(2), 247-256.
Dillman, L. M. (2012). Evaluator skill acquisition: Linking educational experiences to competencies. American Journal of Evaluation, 34(2), 270-285.
Ghere, G., King, J. A., Stevahn, L., & Minnema, J. (2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27(1), 108-123.
Giles, D. E. Jr. & Eyler, J. (1994, Fall). The theoretical roots of service-learning in John Dewey: Toward a theory of service-learning. Michigan Journal of Community Service Learning, 77–85.
Author and other. (2017).
Author and other. (2015).
Author. (2010).
Author S. (2009).
Newcomer, K.E., Hatry, H.P., & Wholey, J.S. (4th ed., 2015). Handbook of Practical Program Evaluation, 4th Ed. Jossey-Bass. ISBN: 978-1-118-89360-9
Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Upper Saddle River, NJ: Prentice Hall.
LaVelle, J.M. (2020). Using a Program Evaluation Course to Foster Service and Humanize Students, Faculty, and Community Partners, Sengupta, E., Blessinger, P. and Makhanya, M. (Ed.) Integrating Community Service into Curriculum: International Perspectives on Humanizing Education (Innovations in Higher Education Teaching and Learning, Vol. 25), Emerald Publishing Limited, pp. 129-146. https://doi.org/10.1108/S2055-364120200000025009
Linfield, K. J. (2019). Hands-on program evaluation training: The evolution of a doctoral course. Journal of Prevention & Intervention in the Community, 1-17. https://doi.org/ 10.1080/10852352.2019.1643580
Levin-Rozalis, M., & Rosenstein, B. (2003). A mentoring approach to the one-year evaluation course. American Journal of Evaluation, 24(2), 245-259.
Lovat, T., & Clement, N. (2016). Service Learning as Holistic Values Pedagogy. Journal of Experiential Learning, 39(2), 115-129. https://doi.org/10.1177/1053828548
McDavid, JC, Huse, I., Hawthorn, L.R.L. (2019). Program Evaluation and Performance Measurement – An introduction to practice (3rd Ed.) Sage Publication, Inc.
Mertens, D., and Wilson, A. (2012). Program evaluation theory and practice: A comprehensive guide. The Guilford Press. ISBN:9781462503155
Novak, J.M., Markey, V., & Allen, M. (2007). Evaluating cognitive outcomes of service learning in higher education: A meta-analysis. Communication Research Reports, 24(2), 149-157.
Poth, CN, Searle M, Aquilina AM, Ge J, & Elder A. (2020). Assessing competency-based evaluation course impacts: A mixed methods case study. Evaluation and Program Planning 79, 101789. Retrieved Jan. 30, 2020, from https://doi.org/10.1016/j.evalprogplan.2020.101789.
Ramirez, R., Brodhead, D., & Quarry, W. (2018). Readiness in evaluation: Three prompts for evaluators. Canadian Journal of Program Evaluation, 33(2), 258-267. https://doi.org/10.3138/cjpe.42238
Ridde, V., Fournier, P., Banza, B., Tourigny, C., & Ouédraogo, D. (2009). Programme evaluation training for health professionals in francophone Africa: Process, competence acquisition and use. Human Resources for Health, 7, 3. https://doi.org/10.1186/1478-4491-7-3.
Rossi, P., Lipsey, M., & Freeman, H. (2003). Evaluation: A systematic approach. 7th edition. SAGE Publications. ISBN:978076190894
Stevahn, L, King, J., Ghere, G., & Minnema, J.(2005) Establishing Essential Competencies for Program Evaluators. American Journal of Evaluation, 26(1) 43-59. https://wmich.edu/sites/default/files/attachments/u58/2015/Establishing_Essential_Competencies.pdf
The Canadian Evaluation Society Competencies for Canadian Evaluation Practice. (2018 Edition). Canadian Evaluation Society. Retrieved May 23, 2020, from https://evaluationcanada.ca/competencies-canadian-evaluators.
Tyng, CM, Amin, HU, Saad, MN, and Malik, AS (2017). The influences of emotion on learning and memory. Frontiers in Psychology, 8,1454. https://doi.org/10.3389/fpsyg.2017.01454
AEA Evaluator Competencies (Revised April 5, 2018). American Evaluation Association. Retrieved Oct. 21, 2020, from https://www.eval.org/page/competencies.
Association of Experiential Education (n.d.). What is Experiential Education? Retrieved Nov. 10th, 2020, from http://www.aee.org/what-is-ee.
Bakken, L. L., Núñez, J., & Couture, C. (2014). A course model for building evaluation capacity through a University–community partnership. American Journal of Evaluation, 35(4), 579-593.
Centers for Disease Control and Prevention. (2011). Office of the Director, Office of Strategy and Innovation. U.S. Department of Health and Human Services. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention.
Cauley, K., Canfield, A., Clasen, C., Dobbins, J., Hemphill, S., Jaballas, E., et al. (2001). Service learning: Integrating student learning and community service. Education for Health, 14(2), 173-181.
Conway, J.M., Amel, E.L., & Gerwien, D.P. Teaching and learning in the social context: A meta-analysis of service learning’s effects on academic, personal, social, and citizenship outcomes. (2009). Teaching of Psychology, 36: 233-245. https://doi.org/10.1080/00986280903172969
Christie, C. A., Quiñones, P., & Fierro, L. (2014). Informing the discussion on evaluator training: A look at evaluators’ course taking and professional practice. American Journal of Evaluation, 32 (2), 274–290.
Creswell J. (2016). 30 essential skills for the qualitative researcher. Sage Publications, Inc.
Creswell, J., & Clark V.L. (2018). Designing and conducting mixed methods research. (3rd ed.) Sage publications, Inc. ISBN: 978-1-4833-4437-9
Darabi, A. (2002). Teaching program evaluation: Using a systems approach. American Journal of Evaluation, 23(2), 219-228.
Davis, M. V. (2006). Teaching practical public health evaluation methods. American Journal of Evaluation, 27(2), 247-256.
Dillman, L. M. (2012). Evaluator skill acquisition: Linking educational experiences to competencies. American Journal of Evaluation, 34(2), 270-285.
Ghere, G., King, J. A., Stevahn, L., & Minnema, J. (2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27(1), 108-123.
Giles, D. E. Jr. & Eyler, J. (1994, Fall). The theoretical roots of service-learning in John Dewey: Toward a theory of service-learning. Michigan Journal of Community Service Learning, 77–85.
Author and other. (2017).
Author and other. (2015).
Author. (2010).
Author S. (2009).
Newcomer, K.E., Hatry, H.P., & Wholey, J.S. (4th ed., 2015). Handbook of Practical Program Evaluation, 4th Ed. Jossey-Bass. ISBN: 978-1-118-89360-9
Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Upper Saddle River, NJ: Prentice Hall.
LaVelle, J.M. (2020). Using a Program Evaluation Course to Foster Service and Humanize Students, Faculty, and Community Partners, Sengupta, E., Blessinger, P. and Makhanya, M. (Ed.) Integrating Community Service into Curriculum: International Perspectives on Humanizing Education (Innovations in Higher Education Teaching and Learning, Vol. 25), Emerald Publishing Limited, pp. 129-146. https://doi.org/10.1108/S2055-364120200000025009
Linfield, K. J. (2019). Hands-on program evaluation training: The evolution of a doctoral course. Journal of Prevention & Intervention in the Community, 1-17. https://doi.org/ 10.1080/10852352.2019.1643580
Levin-Rozalis, M., & Rosenstein, B. (2003). A mentoring approach to the one-year evaluation course. American Journal of Evaluation, 24(2), 245-259.
Lovat, T., & Clement, N. (2016). Service Learning as Holistic Values Pedagogy. Journal of Experiential Learning, 39(2), 115-129. https://doi.org/10.1177/1053828548
McDavid, JC, Huse, I., Hawthorn, L.R.L. (2019). Program Evaluation and Performance Measurement – An introduction to practice (3rd Ed.) Sage Publication, Inc.
Mertens, D., and Wilson, A. (2012). Program evaluation theory and practice: A comprehensive guide. The Guilford Press. ISBN:9781462503155
Novak, J.M., Markey, V., & Allen, M. (2007). Evaluating cognitive outcomes of service learning in higher education: A meta-analysis. Communication Research Reports, 24(2), 149-157.
Poth, CN, Searle M, Aquilina AM, Ge J, & Elder A. (2020). Assessing competency-based evaluation course impacts: A mixed methods case study. Evaluation and Program Planning 79, 101789. Retrieved Jan. 30, 2020, from https://doi.org/10.1016/j.evalprogplan.2020.101789.
Ramirez, R., Brodhead, D., & Quarry, W. (2018). Readiness in evaluation: Three prompts for evaluators. Canadian Journal of Program Evaluation, 33(2), 258-267. https://doi.org/10.3138/cjpe.42238
Ridde, V., Fournier, P., Banza, B., Tourigny, C., & Ouédraogo, D. (2009). Programme evaluation training for health professionals in francophone Africa: Process, competence acquisition and use. Human Resources for Health, 7, 3. https://doi.org/10.1186/1478-4491-7-3.
Rossi, P., Lipsey, M., & Freeman, H. (2003). Evaluation: A systematic approach. 7th edition. SAGE Publications. ISBN:978076190894
Stevahn, L, King, J., Ghere, G., & Minnema, J.(2005) Establishing Essential Competencies for Program Evaluators. American Journal of Evaluation, 26(1) 43-59. https://wmich.edu/sites/default/files/attachments/u58/2015/Establishing_Essential_Competencies.pdf
The Canadian Evaluation Society Competencies for Canadian Evaluation Practice. (2018 Edition). Canadian Evaluation Society. Retrieved May 23, 2020, from https://evaluationcanada.ca/competencies-canadian-evaluators.
Tyng, CM, Amin, HU, Saad, MN, and Malik, AS (2017). The influences of emotion on learning and memory. Frontiers in Psychology, 8,1454. https://doi.org/10.3389/fpsyg.2017.01454