Conducting Problem-Based Learning Meta-Analysis: Complexities, Implications, and Best Practices

Main Article Content

Andrew Walker
Heather Leary

Abstract

Almost fifty years ago Barrows claimed that problem-based learning (PBL) was broad enough that a single methodological description was not possible (1986). It has only grown more complex since. In addition to meaningful variations of problem-based learning there are several related problem-centered pedagogies, such as case-based learning, project-based learning, and inquiry-based learning among others. Even within PBL primary research is conducted using a wide variation in measurement approaches, with diverse audiences, in a myriad of disciplines. Some of these challenges are unique and still pair with common meta-analysis challenges such as multiple definitions of “control groups,” pre-experimental designs, and multiple treatment designs. This intersection of complexity and common challenges makes PBL meta-analysis research particularly difficult. This article will explore best practices for conducting meta-analysis including search strategies, inclusion/exclusion criteria, asking clear research questions of the primary source literature, coding schemes, analysis techniques including key decisions of random vs. fixed effects models, analyzing publication bias and interpreting, presenting, and discussing results. Beginning with comprehensive coverage of the PBL meta-analysis and meta-synthesis contributions to date and suggestions for new meta-analysis work in PBL, we will mostly be devoted to the best meta-analysis practices alongside their implications unique to a PBL context. 

Article Details

Section
Special Issue: Research Methodologies for studying PBL

References

Albanese, M. (2000). Problem-based learning: Why curricula are likely to show little effect on knowledge and clinical skills. Medical Education, 34(9), 729–738.

Albanese, M., & Mitchell, S. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues. Academic Medicine, 68(1), 52–81.

Barneveld, A. V., & Strobel, J. (2008). Is PBL effective? A meta-synthesis of meta-analyses comparing problem-based learning to conventional classroom learning.

Barrows, H. S. (1986). A taxonomy of problem-based learning methods. Medical Education, 20(6), 481–486.

Barrows, H. S. (1996). Problem-based learning in medicine and beyond: A brief overview. New Directions for Teaching and Learning, 68, 3–12.

Belland, B., Walker, A., & Kim, N. (2017). A Bayesian Network Meta-Analysis to Synthesize the Influence of Contexts of Scaffolding Use on Cognitive Outcomes in STEM Education. Review of Educational Research, 87(6), 1042–1081.

Belland, B., Walker, A., Kim, N., & Lefler, M. (2017). Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education: A Meta-Analysis. Review of Educational Research, 82(2), 309–344.

Berkson, L. (1993). Problem-based learning: Have the expectations been met? Academic Medicine, 68(10 Suppl.), S79-88.

Boote, D. N., & Beile, P. (2005). Scholars Before Researchers: On the Centrality of the Dissertation Literature Review in Research Preparation. Educational Researcher, 34(6), 3–15. https://doi.org/10.3102/0013189X034006003

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021). Introduction to Meta-Analysis (2nd edition). Wiley.

Campbell Systematic Reviews. (n.d.). Wiley Online Library. https://doi.org/10.1002/(ISSN)1891-1803

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Earlbaum Associates.

Colliver, J. A. (2000). Effectiveness of problem-based learning curricula: Research and theory. Academic Medicine, 75(3), 259–266.

Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2019). The Handbook of Research Synthesis and Meta-Analysis (3rd edition). Russell Sage Foundation.

Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: A meta-analysis. Learning and Instruction : The Journal of the European Association for Research on Learning and Instruction, 13(5), 533–568.

Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629

Gijbels, D., Dochy, F., Van den Bossche, P., & Segers, M. (2005). Effects of problem-based learning: A meta-analysis from the angle of assessment. Review of Educational Research, 75, 27–61. https://doi.org/10.3102/00346543075001027

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8.

Glass, G. V. (2006). Meta-Analysis: The quantitative synthesis of research findings. In J. L. Green, G. Camilli, & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 427–438). Lawrence Erlbaum Associates, Publishers.

Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181–217. https://doi.org/10.1002/jrsm.1378

Hansen, C., Steinmetz, H., & Block, J. (2022). How to conduct a meta-analysis in eight steps: A practical guide. Management Review Quarterly, 72, 1-19. https://doi.org/10.1007/s11301-021-00247-4

Harari, M. B., Parola, H. R., Hartwell, C. J., & Riegelman, A. (2020). Literature searches in systematic reviews and meta-analyses: A review, evaluation, and recommendations. Journal of Vocational Behavior, 118, 1-11. https://doi.org/10.1016/j.jvb.2020.103377

Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65. https://doi.org/10.1002/jrsm.5

Hmelo-Silver, C. E., Duncan, R. G., & Chinn, C. A. (2007). Scaffolding and Achievement in Problem-Based and Inquiry Learning: A Response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 99–107. https://doi.org/10.1080/00461520701263368

Jonassen, D. H. (2000). Toward a Design Theory of Problem Solving. Educational Technology Research and Development, 48(4), 63–85.

Kalaian, H. A., Mullan, P. B., & Kasim, R. M. (1999). What can studies of problem-based learning tell us? Synthesizing and modeling PBL effects on National Board of medical Examination Performance: Hierarchical linear modeling meta-analytic approach. Advances in Health Sciences Education, 4(3), 209–221.

Kraft, M. A. (2020). Interpreting Effect Sizes of Education Interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798

Authors (2018). Meta-Analysis and Meta-Synthesis Methodologies: Rigorously Piecing Together Research. TechTrends: Linking Research and Practice to Improve Learning, 62(5), 525–534. https://doi.org/10.1007/s11528-018-0312-7

Authors (2013). Exploring the Relationships Between Tutor Background, Tutor Training, and Student Learning: A Problem-based Learning Meta-Analysis. Interdisciplinary Journal of Problem-Based Learning, 7(1). https://doi.org/10.7771/1541-5015.1331

Authors (2022). Forms of Computer-Based Scaffolding in Engineering Education: A Meta-Analysis. Association for Educational Communications and Technology, Las Vegas, NV.

Lipsey, M., & Wilson, D. (2001). Practical meta-analysis. Sage.

Newman, M. (2003). A Pilot Systematic Review and Meta-Analysis on the Effectiveness of Problem Based Learning.

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641. https://doi.org/10.1037/0033-2909.86.3.638

Steel, P., Beugelsdijk, S., & Aguinis, H. (2021). The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews. Journal of International Business Studies, 52(1), 23–44.

Sugrue, B. (1993). Specifictions for the design of problem-solving assessments in science (84.117G). National Center for Educational Research and Improvement.

Sugrue, B. (1995). A theory-based framework for assessing domain-specific problem solving ability. Educational Measurement: Issues and Practice, 14(3), 29–36.

Vernon, D. T., & Blake, R. L. (1993). Does Problem-based Learning Work? A Meta-analysis of Evaluative Research. Academic Medicine, 68(7), 550–563.

Authors (2009). A Problem-Based Learning Meta Analysis: Differences Across Problem Types, Implementation Types, Disciplines, and Assessment Levels. Interdisciplinary Journal of Problem-Based Learning, 3(1), 12–43.

Wilson, D. (n.d.). Practical Meta-Analysis Effect Size Calculator. Retrieved January 3, 2023, from https://www.campbellcollaboration.org/escalc/html/EffectSizeCalculator-R-main.php