Conducting Problem-Based Learning Meta-Analysis: Complexities, Implications, and Best Practices
Article Sidebar
Main Article Content
Abstract
Almost fifty years ago Barrows claimed that problem-based learning (PBL) was broad enough that a single methodological description was not possible (1986). It has only grown more complex since. In addition to meaningful variations of problem-based learning there are several related problem-centered pedagogies, such as case-based learning, project-based learning, and inquiry-based learning among others. Even within PBL primary research is conducted using a wide variation in measurement approaches, with diverse audiences, in a myriad of disciplines. Some of these challenges are unique and still pair with common meta-analysis challenges such as multiple definitions of “control groups,” pre-experimental designs, and multiple treatment designs. This intersection of complexity and common challenges makes PBL meta-analysis research particularly difficult. This article will explore best practices for conducting meta-analysis including search strategies, inclusion/exclusion criteria, asking clear research questions of the primary source literature, coding schemes, analysis techniques including key decisions of random vs. fixed effects models, analyzing publication bias and interpreting, presenting, and discussing results. Beginning with comprehensive coverage of the PBL meta-analysis and meta-synthesis contributions to date and suggestions for new meta-analysis work in PBL, we will mostly be devoted to the best meta-analysis practices alongside their implications unique to a PBL context.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
1. Publication and Promotion: In consideration of the Publisher’s agreement to publish the Work, Author hereby grants and assigns to Publisher the non-exclusive right to print, publish, reproduce, or distribute the Work throughout the world in all means of expression by any method now known or hereafter developed, including electronic format, and to market or sell the Work orany part of it as Publisher sees fit. Author further grants Publisher the right to use Author’s name in association with the Work inpublished form and in advertising and promotional materials
2. Copyright: Copyright of the Work remains in Author’s name.
3. Prior Publication and Attribution: Author agrees not to publish the Work in print form prior to publication of the Work by the Publisher. Author agrees to cite, by author, title, and publisher, the original Interdisciplinary Journal of Problem-based Learning publication when publishing the Work elsewhere
4. Author Representations: The Author represents and warrants that the Work:
(a) is the Author’s original Work and that Author has full power to enter into this Agreement;
(b) does not infringe the copyright or property of another;
(c) contains no material which is obscene, libelous, defamatory or previously published, in whole or in part.
Author shall indemnify and hold Publisher harmless against loss of expenses arising from breach of any such warranties.
5. Licensing and Reuse: Reuse of the published Work will be governed by a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0; http://creativecommons.org/licenses/ by-nc/4.0/). This license lets others remix, tweak, and build upon the Work non-commercially; although new works must acknowledge the original Interdisciplinary Journal of Problem-based Learning publication and be non-commercial, they do not have to be licensed on the same terms.
References
Albanese, M. (2000). Problem-based learning: Why curricula are likely to show little effect on knowledge and clinical skills. Medical Education, 34(9), 729–738.
Albanese, M., & Mitchell, S. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues. Academic Medicine, 68(1), 52–81.
Barneveld, A. V., & Strobel, J. (2008). Is PBL effective? A meta-synthesis of meta-analyses comparing problem-based learning to conventional classroom learning.
Barrows, H. S. (1986). A taxonomy of problem-based learning methods. Medical Education, 20(6), 481–486.
Barrows, H. S. (1996). Problem-based learning in medicine and beyond: A brief overview. New Directions for Teaching and Learning, 68, 3–12.
Belland, B., Walker, A., & Kim, N. (2017). A Bayesian Network Meta-Analysis to Synthesize the Influence of Contexts of Scaffolding Use on Cognitive Outcomes in STEM Education. Review of Educational Research, 87(6), 1042–1081.
Belland, B., Walker, A., Kim, N., & Lefler, M. (2017). Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education: A Meta-Analysis. Review of Educational Research, 82(2), 309–344.
Berkson, L. (1993). Problem-based learning: Have the expectations been met? Academic Medicine, 68(10 Suppl.), S79-88.
Boote, D. N., & Beile, P. (2005). Scholars Before Researchers: On the Centrality of the Dissertation Literature Review in Research Preparation. Educational Researcher, 34(6), 3–15. https://doi.org/10.3102/0013189X034006003
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021). Introduction to Meta-Analysis (2nd edition). Wiley.
Campbell Systematic Reviews. (n.d.). Wiley Online Library. https://doi.org/10.1002/(ISSN)1891-1803
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Earlbaum Associates.
Colliver, J. A. (2000). Effectiveness of problem-based learning curricula: Research and theory. Academic Medicine, 75(3), 259–266.
Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2019). The Handbook of Research Synthesis and Meta-Analysis (3rd edition). Russell Sage Foundation.
Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: A meta-analysis. Learning and Instruction : The Journal of the European Association for Research on Learning and Instruction, 13(5), 533–568.
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629
Gijbels, D., Dochy, F., Van den Bossche, P., & Segers, M. (2005). Effects of problem-based learning: A meta-analysis from the angle of assessment. Review of Educational Research, 75, 27–61. https://doi.org/10.3102/00346543075001027
Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8.
Glass, G. V. (2006). Meta-Analysis: The quantitative synthesis of research findings. In J. L. Green, G. Camilli, & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 427–438). Lawrence Erlbaum Associates, Publishers.
Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181–217. https://doi.org/10.1002/jrsm.1378
Hansen, C., Steinmetz, H., & Block, J. (2022). How to conduct a meta-analysis in eight steps: A practical guide. Management Review Quarterly, 72, 1-19. https://doi.org/10.1007/s11301-021-00247-4
Harari, M. B., Parola, H. R., Hartwell, C. J., & Riegelman, A. (2020). Literature searches in systematic reviews and meta-analyses: A review, evaluation, and recommendations. Journal of Vocational Behavior, 118, 1-11. https://doi.org/10.1016/j.jvb.2020.103377
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65. https://doi.org/10.1002/jrsm.5
Hmelo-Silver, C. E., Duncan, R. G., & Chinn, C. A. (2007). Scaffolding and Achievement in Problem-Based and Inquiry Learning: A Response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 99–107. https://doi.org/10.1080/00461520701263368
Jonassen, D. H. (2000). Toward a Design Theory of Problem Solving. Educational Technology Research and Development, 48(4), 63–85.
Kalaian, H. A., Mullan, P. B., & Kasim, R. M. (1999). What can studies of problem-based learning tell us? Synthesizing and modeling PBL effects on National Board of medical Examination Performance: Hierarchical linear modeling meta-analytic approach. Advances in Health Sciences Education, 4(3), 209–221.
Kraft, M. A. (2020). Interpreting Effect Sizes of Education Interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798
Authors (2018). Meta-Analysis and Meta-Synthesis Methodologies: Rigorously Piecing Together Research. TechTrends: Linking Research and Practice to Improve Learning, 62(5), 525–534. https://doi.org/10.1007/s11528-018-0312-7
Authors (2013). Exploring the Relationships Between Tutor Background, Tutor Training, and Student Learning: A Problem-based Learning Meta-Analysis. Interdisciplinary Journal of Problem-Based Learning, 7(1). https://doi.org/10.7771/1541-5015.1331
Authors (2022). Forms of Computer-Based Scaffolding in Engineering Education: A Meta-Analysis. Association for Educational Communications and Technology, Las Vegas, NV.
Lipsey, M., & Wilson, D. (2001). Practical meta-analysis. Sage.
Newman, M. (2003). A Pilot Systematic Review and Meta-Analysis on the Effectiveness of Problem Based Learning.
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641. https://doi.org/10.1037/0033-2909.86.3.638
Steel, P., Beugelsdijk, S., & Aguinis, H. (2021). The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews. Journal of International Business Studies, 52(1), 23–44.
Sugrue, B. (1993). Specifictions for the design of problem-solving assessments in science (84.117G). National Center for Educational Research and Improvement.
Sugrue, B. (1995). A theory-based framework for assessing domain-specific problem solving ability. Educational Measurement: Issues and Practice, 14(3), 29–36.
Vernon, D. T., & Blake, R. L. (1993). Does Problem-based Learning Work? A Meta-analysis of Evaluative Research. Academic Medicine, 68(7), 550–563.
Authors (2009). A Problem-Based Learning Meta Analysis: Differences Across Problem Types, Implementation Types, Disciplines, and Assessment Levels. Interdisciplinary Journal of Problem-Based Learning, 3(1), 12–43.
Wilson, D. (n.d.). Practical Meta-Analysis Effect Size Calculator. Retrieved January 3, 2023, from https://www.campbellcollaboration.org/escalc/html/EffectSizeCalculator-R-main.php