Cruel and Usual The Psychological and Financial Cost of SETs
Article Sidebar
Main Article Content
Abstract
In higher education, student evaluation of teaching (SET) has been under scrutiny for its lack of validity, weak correlation with student learning, and bias toward historically marginalized faculty. Absent from the literature are the psychological and financial implications of negative SETs. In an exploratory mixed methods study of 344 instructors, our findings reveal that the majority of respondents have been impacted both psychologically and financially. Female faculty and faculty of color are disproportionately affected psychologically with female faculty reporting negative affective experiences for 5 to 20+ years. Common reasons include feelings of powerlessness, frustrations with SETs as a metric for teacher quality, and biases linked to instructors' identities. Financially, both female faculty and faculty of color were more likely to report measurable and immeasurable financial losses, ranging from denial of promotion to additional time for student care or decreased time to advance one’s career. The pressure on historically marginalized faculty to counter bias in SETs has wide-ranging psychological and financial repercussions, which underscores the labor inequities of an unjust system of assessment and the ethical implications of universities and colleges requiring faculty to review biased, and at times, abusive student comments.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
- Authors retain copyright and grant the Journal of the Scholarship of Teaching and Learning (JoSoTL) right of first publication with the work simultaneously licensed under a Creative Commons Attribution License, (CC-BY) 4.0 International, allowing others to share the work with proper acknowledgement and citation of the work's authorship and initial publication in the Journal of the Scholarship of Teaching and Learning.
- Authors are able to enter separate, additional contractual agreements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in the Journal of the Scholarship of Teaching and Learning.
- In pursuit of manuscripts of the highest quality, multiple opportunities for mentoring, and greater reach and citation of JoSoTL publications, JoSoTL encourages authors to share their drafts to seek feedback from relevant communities unless the manuscript is already under review or in the publication queue after being accepted. In other words, to be eligible for publication in JoSoTL, manuscripts should not be shared publicly (e.g., online), while under review (after being initially submitted, or after being revised and resubmitted for reconsideration), or upon notice of acceptance and before publication. Once published, authors are strongly encouraged to share the published version widely, with an acknowledgement of its initial publication in the Journal of the Scholarship of Teaching and Learning.
References
References
Aitkaliyeva, A., Ishoy, J., Jasser-Stone, J., Wharry, J., Lawson-Ross, A., & Hunyadi Murph, S.,E. (2021). Initiating Mental Health Conversations in Academia. Jom, 73(8), 2260-2263. https://www.proquest.com/scholarly-journals/initiating-mental-health-conversations-academia/docview/2555433670/se-2
American Association of University Professors (2018). Data snapshot: Contingent faculty in US higher ed. https://www.aaup.org/news/data-snapshot-contingent-faculty-us-higher-ed#.XdBrxS2ZOYU
Basow, S., Codos, S., & Martin, J. (2013). The effects of professors' race and gender on student evaluations and performance. College Student Journal, 47(2), 352-363.
Beran, T. N., & Rokosh, J. L. (2009). The consequential validity of student ratings: What do instructors really think? Alberta Journal of Educational Research, 55(4).
Boring, A. (2017). Gender biases in student evaluations of teaching. Journal of Public Economics, 145, 27–41. https://doi.org/10.1016/j.jpubeco.2016.11.006
Boring, A., Ottoboni, K., & Stark, P. (2016). Student evaluations of teaching (mostly) do not measure teaching effectiveness. ScienceOpen Research. https://doi.org/10.14293/s2199-1006.1.sor-edu.aetbzc.v1
Bruce, R. F., & Wiebe, J. M. (1999). Recognizing subjectivity: A case study in manual tagging. Natural Language Engineering, 5(2), 187-205.
Burke, A. S., Head-Burgess, W., & Siders, M. (2017). He’s smart and she’s nice: Student perceptions of male and female faculty. International Journal of Gender and Women’s Studies, 5(1), 1-6.
Buser, W., Batz-Barbarich, C. L., & Hayter, J. K. (2022). Evaluation of women in economics: Evidence of gender bias following behavioral role violations. Sex Roles, 86(11-12), 695-710.
Carpenter, S. K., Witherby, A. E., & Tauber, S. K. (2020). On students’ (mis)judgments of learning and teaching effectiveness: Where we stand and how to move forward. Journal of Applied Research in Memory and Cognition, 9(2), 181-185. https://doi.org/10.1016/j.jarmac.2020.04.003
Chatman, J. A., Sharps, D., Mishra, S., Kray, L. J., & North, M. S. (2022). Agentic but not warm: Age-gender interactions and the consequences of stereotype incongruity perceptions for middle-aged professional women. Organizational Behavior and Human Decision Processes, 173, 104190.
Chávez, K., & Mitchell, K. M. (2020). Exploring bias in student evaluations: Gender, race, and ethnicity. PS: Political Science & Politics, 53(2), 270-274.
Clayson, D. E. (2009). Student evaluations of teaching: Are they related to what students learn? A meta-analysis and review of the literature. Journal of marketing education, 31(1), 16-30.
Clayson, D. (2022). The student evaluation of teaching and likability: What the evaluations actually measure. Assessment & Evaluation in Higher Education, 47(2), 313–326. https://doi-org.ezproxy.ithaca.edu/10.1080/02602938.2021.1909702
Clayson, D. E., & Haley, D. A. (2011). Are students telling us the truth? A critical look at the student evaluation of teaching. Marketing Education Review, 21(2), 101–112. https://doi.org/10.2753/MER1052-8008210201
Colby, G., & Fowler, C. (2020). Data snapshot: IPEDS data on full-time women faculty and faculty of color. American Association of University Professors. https://www.aaup.org/sites/default/files/Dec-2020_Data_Snapshot_Women_and_Faculty_of_Color.pdf
Creswell, J. W., & Creswell, D.J. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Thousand Oaks, CA: Sage.
Cunningham, S., Laundon, M., Cathcart, A., Bashar, M. A., & Nayak, R. (2023). First, do no harm: Automated detection of abusive comments in student evaluation of teaching surveys. Assessment & Evaluation in Higher Education, 48(3), 377-389.
El-Alayli, A., Hansen-Brown, A. A., & Ceynar, M. (2018). Dancing backwards in high heels: Female professors experience more work demands and special favor requests, particularly from academically entitled students. Sex Roles, 79, 136-150.
Eskreis-Winkler, L., & Fishbach, A. (2022). You think failure is hard? So is learning from it. Perspectives on Psychological Science, 17(6), 1511-1524.
Flaherty, C. (2016, January 11). Bias against female instructors. Inside Higher Ed. https://www.insidehighered.com/news/2016/01/11/new-analysis-offers-more-evidence-against-student-evaluations-teaching
Flaherty, C. (2018, May 22). Teaching eval shake-up. Inside Higher Ed. https://www.insidehighered.com/news/2018/05/22/most-institutions-say-they-value-teaching-how-they-assess-it-tells-different-story
Flaherty, C. (2021, February 17). The skinny on teaching evals and bias. Inside Higher Ed.
https://www.insidehighered.com/news/2021/02/17/whats-really-going-respect-bias-and-teaching-evals
Flaherty, C. (2022, October 30). Ratings and gender bias over time. Inside Higher Ed. https://www.insidehighered.com/news/2022/10/31/ratings-and-bias-against-women-over-time
Galbraith, C. S., Merrill, G. B., & Kline, D. M. (2012). Are student evaluations of teaching effectiveness valid for measuring student learning outcomes in business related classes? A neural network and bayesian analyses. Research in Higher Education, 53(3), 353-374.
Heffernan, T. (2022). Sexism, racism, prejudice, and bias: a literature review and synthesis of research surrounding student evaluations of courses and teaching. Assessment & Evaluation in Higher Education, 47(1), 144–154. https://doi-org.ezproxy.ithaca.edu/10.1080/02602938.2021.1888075
Heffernan, T. (2023). Abusive comments in student evaluations of courses and teaching: the attacks women and marginalised academics endure. Higher Education, 85(1), 225–239. https://doi.org/10.1007/s10734-022-00831-x
Heffernan, T., & Harpur, P. (2023): Discrimination against academics and career implications of student evaluations: University policy versus legal compliance. Assessment & Evaluation in Higher Education, DOI: 10.1080/02602938.2023.2225806
Heilman, M. E. (2012). Gender stereotypes and workplace bias. Research in Organizational Behavior, 32, 113–135. https://doi.org/10.1016/j.riob.2012.11.003
Hutchinson, M., Coutts, R., Massey, D., Nasrawi, D., Fielden, J., Lee, M., & Lakeman, R. (2023). Student evaluation of teaching: reactions of Australian academics to anonymous non-constructive student commentary. Assessment and Evaluation in Higher Education, 1–11. https://doi.org/10.1080/02602938.2023.2195598
Keng, S. H. (2018). Tenure system and its impact on grading leniency, teaching effectiveness and student effort. Empirical Economics, 55(3), 1207–1227. https://doi-org.ezproxy.ithaca.edu/10.1007/s00181-017-1313-7
Kinman, G. (2016). Effort-reward imbalance and overcommitment in UK academics: Implications
for mental health, satisfaction and retention. Journal of Higher Education Policy and Management, 38(5), 504-518. https://doi.org/10.1080/1360080X.2016.1181884
Kinman, G., & Wray, S. (2015). Work-related wellbeing in UK higher education-2014. University
and College Union. DOI: 10.13140/RG.2.2.24867.76321
Kornell, N., & Hausman, H. (2016). Do the best teachers get the best ratings? Frontiers in Psychology,
https://doi.org/10.3389/fpsyg.2016.00570
Kreitzer, R. J., & Sweet-Cushman, J. (2022). Evaluating student evaluations of teaching: A review of
measurement and equity bias in SETs and recommendations for ethical reform. Journal of Academic Ethics, 20(1), 73–84. https://doi.org/10.1007/s10805-021-09400-w
Lakeman, R., Coutts, R., Hutchinson, M., Lee, M., Massey, D., Nasrawi, D., & Fielden, J. (2022a).
Appearance, insults, allegations, blame and threats: An analysis of anonymous non-
constructive student evaluation of teaching in Australia. Assessment & Evaluation in Higher Education, 47(8), 1245-1258.
Lakeman, R., Coutts, R., Hutchinson, M., Massey, D., Nasrawi, D., Fielden, J., & Lee, M. (2022b). Stress, distress, disorder and coping: The impact of anonymous student evaluation of teaching on the health of higher education teachers. Assessment & Evaluation in Higher Education, 47(8), 1489-1500.
Mengel, F., Sauermann, J., & Zölitz, U. (2019). Gender bias in teaching evaluations. Journal of the European Economic Association, 17(2), 535–566. https://doi-org.ezproxy.ithaca.edu/10.1093/jeea/jvx057
Mitchell, K. M., & Martin, J. (2018). Gender bias in student evaluations. PS: Political Science & Politics, 51(3), 648-652.
Mun, C. (2020). The many harms of SETs in higher education. Symposion: Theoretical and Applied Inquiries in Philosophy and Social Sciences, 7(2), 285–314. https://doi.org/10.5840/symposion20207219
National Center for Education Statistics. (2022). Characteristics of postsecondary faculty. U.S. Department of Education, Institute of Education Sciences. Retrieved June 2, 2023, from https://nces.ed.gov/programs/coe/indicator/csc.
Pittman, C. T. (2010). Race and gender oppression in the classroom: The experiences of women faculty of color with white male students. Teaching Sociology, 38(3), 183-196.
Rosen, A. S. (2018). Correlations, trends and potential biases among publicly accessible web-based student evaluations of teaching: A large-scale study of RateMyProfessors.com data. Assessment and Evaluation in Higher Education, 43(1), 31–44. https://doi.org/10.1080/02602938.2016.1276155
Reid, L. D. (2010). The role of perceived race and gender in the evaluation of college teaching on RateMyProfessors.com. Journal of Diversity in Higher Education, 3(3), 137.
Richardson, J. E. (2007). Analysing newspapers: An approach from critical discourse analysis. Bloomsbury Publishing.
Riniolo, T. C., Johnson, K. C., Sherman, T. R., & Misso, J. A. (2006). Hot or not: Do professors perceived as physically attractive receive higher student evaluations?. The Journal of General Psychology, 133(1), 19-35.
Saldaña, J. (2021). The coding manual for qualitative researchers. 4th ed., SAGE Publishing.
Schmidt, B. (2015). Gendered Language in Teacher Reviews. http://benschmidt.org/profGender/.
Simpson, P. M., & Siguaw, J. A. (2000). Student evaluations of teaching: An exploratory study of the faculty response. Journal of Marketing Education, 22(3), 199-213.
Smith, B. P. (2007). Student ratings of teaching effectiveness: An analysis of end-of-course faculty evaluations. College Student Journal, 41(4).
Smith, B. P., & Hawkins, B. (2011). Examining student evaluations of black college faculty: Does race matter? The Journal of Negro Education, 149-162.
Spooren, P., & Christiaens, W. (2017). I liked your course because I believe in (the power of) student evaluations of teaching (SET). Students’ perceptions of a teaching evaluation process and their relationships with SET scores. Studies in Educational Evaluation, 54, 43-49.
Sprague, J., & Massoni, K. (2005). Student evaluations and gendered expectations: What we can't count can hurt us. Sex Roles, 53(11), 779-793.
Storage, D., Horne, Z., Cimpian, A., & Leslie, S. J. (2016). The frequency of “brilliant” and “genius” in teaching evaluations predicts the representation of women and African Americans across fields. PloS one, 11(3), e0150194.
Stroebe, W. (2016). Why good teaching evaluations may reward bad teaching: On grade inflation and
other unintended consequences of student evaluations. Perspectives on Psychological Science, 11(6), 800-816.
Stroebe, W. (2020) Student evaluations of teaching encourages poor teaching and contributes to
grade inflation: A theoretical and empirical analysis. Basic and Applied Social Psychology, 42(4), 276-294, DOI: 10.1080/01973533.2020.1756817
Taylor, L. U. (Ed.). (2021). Implications of race and racism in student evaluations of teaching: The hate u give. Lexington Books.
Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42. https://doi.org/10.1016/j.stueduc.2016.08.007
Uttl, B. (2021). Lessons learned from research on student evaluation of teaching in higher education. In W., Rollett, H., Bijlsma, H., & S. Röhl (Eds.), Student Feedback on Teaching in Schools: Using Student Perceptions for the Development of Teaching and Teachers (pp. 237-256). Springer Nature. DOI: 10.1007/978-3-030-75150-0
Vasey, C., & Carroll, L. (2016). How do we evaluate teaching? Academe, 102(3), 34–39.
Wallace, S. L., Lewis, A. K., & Allen, M. D. (2019). The state of the literature on student evaluations of teaching and an exploratory analysis of written comments: Who benefits most? College Teaching, 67(1), 1–14. https://doi-org.ezproxy.ithaca.edu/10.1080/87567555.2018.1483317
Wolbring, T., & Riordan, P. (2016). How beauty works. Theoretical mechanisms and two empirical applications on students' evaluation of teaching. Social Science Research, 57, 253-272.
Zheng X, Vastrad S., He, J., & Ni, C. (2023). Contextualizing gender disparities in online teaching evaluations for professors. PLoS ONE 18(3): e0282704. https://doi.org/10.1371/journal.pone.0282704