Bibliometric Review Methodology and State of the Science Bibliometric Review of Research on Problem-based Learning, 2017-2021

Main Article Content

Philip Hallinger


Bibliometric reviews of research have gained increased popularity since the mid-2010s. Yet, many readers may be uncertain as to the purpose and method of bibliometric reviews. This article provides an introduction to bibliometric review methods, clarifying their purpose, as well as methods of data collection, analysis, and interpretation. The latter portion of the article is devoted to a state-of-the-science, bibliometric review of research on problem-based learning. In this review, the author applied descriptive statistics, citation analysis, and co-citation analysis to a database of 5,764 Scopus-indexed documents on problem-based learning published between 2017 and 2021. The analyses found increasing diversity of the PBL knowledge base with respect to both geographical, subject area origins, and focus of the research when compared with findings reported in past reviews. Author co-citation analysis visualized the intellectual structure of this recent PBL literature revealing four dominant schools of thought: Interdisciplinary PBL Theory and Practice, Active Learning, Social and Experiential Learning, PBL Process. This finding highlights the increasing integration of PBL with other methods of experiential and active learning.

Article Details

Special Issue: Research Methodologies for studying PBL


Adams, V., Burger, S., Crawford, K., & Setter, R. (2018). Can you escape? Creating an escape room to facilitate active learning. Journal for Nurses in Professional Development, 34(2), E1-E5.

Adriaanse, L. S., & Rensleigh, C. (2013). Web of Science, Scopus and Google Scholar: A content comprehensiveness comparison. The Electronic Library.

AlRyalat, S. A. S., Malkawi, L. W., & Momani, S. M. (2019). Comparing bibliometric analysis using PubMed, Scopus, and Web of Science databases. JoVE (Journal of Visualized Experiments), (152), e58494.

Argaw Aweke, S., Haile Beyene, B., Ayelaw Beyene, T., & Kuma Shiferaw, G. (2017). The effect of problem based learning (pbl) instruction on students’ motivation and problem solving skills of physics. EURASIA Journal of Mathematics Science and Technology Education, 13(3), 857-871.

Azer, S. A. (2017). Top‐cited articles in problem‐based learning: A bibliometric analysis and quality of evidence assessment. Journal of Dental Education, 81(4), 458-478.

Bales, M. E., Wright, D. N., Oxley, P. R., & Wheeler, T. R. (2020). Bibliometric visualization and analysis software: State of the art, workflows, and best practices. E-commons, Cornell University, Ithaca, NY.

Ballen, C. J., Wieman, C., Salehi, S., Searle, J. B., & Zamudio, K. R. (2017). Enhancing diversity in undergraduate science: Self-efficacy drives performance gains with active learning. CBE—Life Sciences Education, 16(4), ar56.

Berkhout, J. J., Helmich, E., Teunissen, P. W., van der Vleuten, C. P., & Jaarsma, A. D. C. (2018). Context matters when striving to promote active and lifelong learning in medical education. Medical education, 52(1), 34-44.

Börner, K., Chen, C., & Boyack, K. W. (2003). Visualizing knowledge domains. Annual Review of Information Science and Technology, 37(1), 179-255.

Burgess, A., Bleasel, J., Haq, I., Roberts, C., Garsia, R., Robertson, T., & Mellis, C. (2017). Team-based learning (TBL) in the medical curriculum: better than PBL? BMC medical education, 17(1), 1-11.

Chen, C., & Chen, C. (2003). Mapping scientific frontiers. London, UK: Springer-Verlag.

Chen, F., Lui, A. M., & Martinelli, S. M. (2017). A systematic review of the effectiveness of flipped classrooms in medical education. Medical Education, 51(6), 585-597.

Chen, J., Kolmos, A., & Du, X. (2021). Forms of implementation and challenges of PBL in engineering education: a review of literature. European Journal of Engineering Education, 46(1), 90-115.

Cobo, M. J., López‐Herrera, A. G., Herrera‐Viedma, E., & Herrera, F. (2011). Science mapping software tools: Review, analysis, and cooperative study among tools. Journal of the American Society for Information Science and Technology, 62(7), 1382-1402.

Dehghanzadeh, S., & Jafaraghaee, F. (2018). Comparing the effects of traditional lecture and flipped classroom on nursing students' critical thinking disposition: A quasi-experimental study. Nurse Education Today, 71, 151-156.

Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251-19257.

Ding, Y., Chowdhury, G. G., & Foo, S. (2001). Bibliometric cartography of information retrieval research by using co-word analysis. Information Processing & Management, 37(6), 817-842.

Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: A meta-analysis. Learning and instruction, 13(5), 533-568.

Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008). Comparison of PubMed, Scopus, web of science, and Google scholar: strengths and weaknesses. The FASEB Journal, 22(2), 338-342.

Fidan, M., & Tuncel, M. (2019). Integrating augmented reality into problem based learning: The effects on learning achievement and attitude in physics education. Computers & Education, 142, 103635.

Frambach, J. M., Driessen, E. W., Beh, P., & van der Vleuten,C. P. (2014). Quiet or questioning? Students’” discussion behaviors in student-centered education across cultures. Studies in Higher Education, 39(6), 1001–1021.

Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359-375.

Gijbels, D., Dochy, F., Van den Bossche, P., & Segers, M. (2005). Effects of problem-based learning: A meta-analysis from the angle of assessment. Review of Educational Research, 75(1), 27-61.

Gilbert, G. N. (1977). Referencing as persuasion. Social Studies of Science, 7(1), 113-122.

Gmür, M. (2003). Co-citation analysis and the search for invisible colleges: A methodological evaluation. Scientometrics, 57, 27–57.

Gough, D., Oliver, S., & Thomas, J. (Eds.). (2017). An introduction to systematic reviews. Sage.

Guerra, A. (2017). Integration of sustainability in engineering education: Why is PBL an answer? International Journal of Sustainability in Higher Education.

Hew, K. F., & Lo, C. K. (2018). Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Medical Education, 18(1), 1-12.

Hood, W., & Wilson, C. (2001). The literature of bibliometrics, scientometrics, and informetrics. Scientometrics, 52(2), 291-314.

Huang, M. H., & Chang, Y. W. (2011). A study of interdisciplinarity in information science: using direct citation and co-authorship analysis. Journal of Information Science, 37(4), 369-378.

Jabarullah, N. H., & Hussain, H. I. (2019). The effectiveness of problem-based learning in technical and vocational education in Malaysia. Education+ Training, 61(5), 552-567.

Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 1(1), 1-9.

Kim, N. J., Belland, B. R., & Walker, A. E. (2018). Effectiveness of computer-based scaffolding in the context of problem-based learning for STEM education: Bayesian meta-analysis. Educational Psychology Review, 30(2), 397-429.

Klavans, R., & Boyack, K. W. (2017). Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge? Journal of the Association for Information Science and Technology, 68(4), 984-998.

Li, S., Cao, M., & Zhu, X. (2019). Evidence-based practice: Knowledge, attitudes, implementation, facilitators, and barriers among community nurses—Systematic review. Medicine, 98(39).

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. SAGE publications, Inc.

Liu, J., Li, Y., Ruan, Z., Fu, G., Chen, X., Sadiq, R., & Deng, Y. (2015). A new method to construct co-author networks. Physica A: Statistical Mechanics and its Applications, 419, 29-39.

Markscheffel, B., & Schröter, F. (2021). Comparison of two science mapping tools based on software technical evaluation and bibliometric case studies. COLLNET Journal of Scientometrics and Information Management, 15(2), 365-396.

McCain, K. W. (1990). Mapping authors in intellectual space: A technical overview. Journal of the American Society for Information Science (1986-1998), 41(6), 433.

McCain, K. W. (1986). Cocited author mapping as a valid representation of intellectual structure. Journal of the American Society for Information Science, 37(3), 111-122.

Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press.

Mingers, J., & Yang, L. (2017). Evaluating journal quality: A review of journal citation indicators and ranking in business and management. European journal of operational research, 257(1), 323-337.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264-269.

Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: a comparative analysis. Scientometrics, 106(1), 213-228.

Moral-Muñoz, J. A., Herrera-Viedma, E., Santisteban-Espejo, A., & Cobo, M. J. (2020). Software tools for conducting bibliometric analysis in science: An up-to-date review. Profesional de la Información, 29(1).

Morton, D. A., & Colbert‐Getz, J. M. (2017). Measuring the impact of the flipped anatomy classroom: The importance of categorizing an assessment by Bloom's taxonomy. Anatomical Sciences Education, 10(2), 170-175.

Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18(1), 1-7.

Njie-Carr, V. P., Ludeman, E., Lee, M. C., Dordunoo, D., Trocky, N. M., & Jenkins, L. S. (2017). An integrative review of flipped classroom teaching models in nursing education. Journal of Professional Nursing, 33(2), 133-144.

Norman, G. T., & Schmidt, H. G. (1992). The psychological basis of problem-based learning: A review of the evidence. Academic Medicine, 67(9), 557-565.

Norris, M., & Oppenheim, C. (2007). Comparing alternatives to the Web of Science for coverage of the social sciences’ literature. Journal of Informetrics, 1(2), 161-169.

Peters, H., & Van Raan, A. (1991). Structuring scientific activities by co-author analysis: An expercise on a university faculty level. Scientometrics, 20(1), 235-255.

Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & McEwen, S. A. (2014). A scoping review of scoping reviews: Advancing the approach and enhancing the consistency. Research Synthesis Methods, 5(4), 371-385.

Pinho, L. A. D., Mota, F. B., Conde, M. V. F., Alves, L. A., & Lopes, R. M. (2015). Mapping knowledge produced on problem-based learning between 1945 and 2014: A bibliometric analysis. Creative Education, 6, 576-584

Price, D. D. S. (1976). A general theory of bibliometric and other cumulative advantage processes. Journal of the American society for Information science, 27(5), 292-306.

Radhakrishnan, S., Erbis, S., Isaacs, J. A., & Kamarthi, S. (2017). Novel keyword co-occurrence network-based methods to foster systematic reviews of scientific literature. PloS One, 12(3), e0172778.

Ramnanan, C. J., & Pound, L. D. (2017). Advances in medical education and practice: student perceptions of the flipped classroom. Advances in Medical Education and Practice, 8, 63.

Reimschisel, T., Herring, A. L., Huang, J., & Minor, T. J. (2017). A systematic review of the published literature on team-based learning in health professions education. Medical Teacher, 39(12), 1227-1237.

Sampaio, R. B., Fonseca, M. V. D. A., & Zicker, F. (2016). Co-authorship network analysis in health research: Method and potential use. Health Research Policy and Systems, 14(1), 1-10.

Schmidt, H. G., Rotgans, J. I., & Yew, E. H. (2011). The process of problem‐based learning: What works and why. Medical Education, 45(8), 792-806.

Small, H. (1973). Co‐citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for information Science, 24(4), 265-269.

Strobel, J., & Van Barneveld, A. (2009). When is PBL more effective? A meta-synthesis of meta-analyses comparing PBL to conventional classrooms. Interdisciplinary Journal of Problem-Based Learning, 3(1), 44-58.

Strotmann, A., & Zhao, D. (2012). Author name disambiguation: What difference does it make in author‐based citation analysis? Journal of the American Society for Information Science and Technology, 63(9), 1820-1833.

Tang, F., Chen, C., Zhu, Y., Zuo, C., Zhong, Y., Wang, N., ... & Liang, D. (2017). Comparison between flipped classroom and lecture-based classroom in ophthalmology clerkship. Medical Education Online, 22(1), 1395679.

Theobald, E. J., Hill, M. J., Tran, E., Agrawal, S., Arroyo, E. N., Behling, S., ... & Freeman, S. (2020). Active learning narrows achievement gaps for underrepresented students in undergraduate science, technology, engineering, and math. Proceedings of the National Academy of Sciences, 117(12), 6476-6483.

Topalli, D., & Cagiltay, N. E. (2018). Improving programming skills in engineering education through problem-based game projects with Scratch. Computers & Education, 120, 64-74.

Trujillo, C. M., & Long, T. M. (2018). Document co-citation analysis to enhance transdisciplinary research. Science Advances, 4(1), e1701130.

Van Eck, N., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, 84(2), 523-538.

Van Eck, N., & Waltman, L. (2018). VOSviewer manual. Centre for Science and Technology Studies. Leiden University. Leiden.

Van Eck, N., & Waltman, L. (2020). VOSviewer version 1.6.8. Centre for Science and Technology Studies. Leiden University. Leiden.

Van Eck, N.J., Waltman, L., Dekker, R. & van den Berg, J. (2010). A comparison of two techniques for bibliometric mapping: Multidimensional scaling and VOS. Journal of the American Society for Information Science and Technology, 61(12), 2405-2416.

Walker, A., Bridges, E., & Chan, B. (1996). Wisdom gained, wisdom given: instituting PBL in a Chinese culture. Journal of Educational Administration, 34(5),12-31

White, H. D., & McCain, K. W. (1998). Visualizing a discipline: An author co‐citation analysis of information science, 1972–1995. Journal of the American Society for Information Science, 49(4), 327-355.

Wallin, J. A. (2005). Bibliometric methods: Pitfalls and possibilities. Basic & Clinical Pharmacology & Toxicology, 97(5), 261-275.

Whittemore, R., & Knafl, K. (2005). The integrative review: updated methodology. Journal of Advanced Nursing, 52(5), 546-553.

Xian, H. & Madhavan, K. (2013). Building on and honoring forty years of PBL scholarship from Howard Barrows: A scientometric, large-scale data, and visualization-based analysis, Interdisciplinary Journal of Problem-Based Learning, 7(1), 10–28.

Zupic, I., & Čater, T. (2015). Bibliometric methods in management and organization. Organizational Research Methods, 18(3), 429-472.