A Multilevel Analysis of Problem-Based Learning Design Characteristics

Main Article Content

Kimberly S. Scott


The increasing use of experience-centered approaches like problem-based learning (PBL) by learning and development practitioners and management educators has raised interest in how to design, implement and evaluate PBL in that field. Of particular interest is how to evaluate the relative impact of design characteristics that exist at the individual and team levels of analysis. This study proposes and tests a multilevel model of PBL design characteristics. Participant perceptions of PBL design characteristics are used to examine PBL reactions and perceived learning outcomes. Findings affirm the importance of problem design characteristics and effective team facilitation, while raising new questions about team-level characteristics such as goal orientation diversity.

Article Details

Research Articles


Araz, G., & Sungur, S. (2007). The interplay between cognitive and motivational variables in a problem-based learning environment. Learning and Individual Differences, 17(4), 291–297. http://dx.doi.org/10.1016/j.lindif.2007.04.003

Armstrong, S. J., & Mahmud, A. (2008). Experiential learning and the acquisition of managerial tacit knowledge. Academy of Management Learning and Education, 7(2), 189– 208. http://dx.doi.org/10.5465/AMLE.2008.32712617

Barrows, H. S., & Tamblyn, R. M. (1980). Problem-based learning: An approach to medical education. New York, NY: Springer.

Barrows, H. S., & Mitchell, D. L. M. (1975). An innovative course in undergraduate neuroscience: Experiment in problem-based learning with “problem boxes.” British Journal of Medical Education, 9(4), 223–230. http://dx.doi.org/10.1111/j.1365-2923.1975.tb01930.x

Benjamin, B., & O’Reilly, C. (2011). Becoming a leader: Early career challenges faced by MBA graduates. Academy of Management Learning and Education, 10(3), 452–472. http://dx.doi.org/10.5465/amle.2011.0002

Bigelow, J. D. (2004). Using problem-based learning to develop skills in solving unstructured problems. Journal of Management Education, 28(5), 591–609. http://dx.doi.org/10.1177/1052562903257310

Breugst, N., Patzelt, H., Shepherd, D. A., & Aguinis, H. (2012). Relationship conflict improves team performance assessment accuracy: Evidence from a multilevel study. Academy of Management Learning and Education, 11(2), 187–206. http://dx.doi.org/10.5465/amle.2011.0032

Brownell, J., & Jameson, D. A. (2004). Problem-based learning in graduate management education: An integrative model and interdisciplinary application. Journal of Management Education, 28(5), 558–577. http://dx.doi.org/10.1177/1052562904266073

Button, S. B., Mathieu, J. E., & Zajac, D. M. (1996). Goal orientation in organizational research: A conceptual and empirical foundation. Organizational Behavior and Human Decision Processes, 67(1), 26–48. http://dx.doi.org/10.1006/obhd.1996.0063

Chaharbaghi, K., & Cox, R. (1995). Problem-based learning: Potential and implementation issues. British Journal of Management, 6(4), 249–256. http://dx.doi.org/10.1111/j.1467-8551.1995.tb00098.x

DeGeest, D., & Brown, K. G. (2011). The role of goal orientation in leadership development. Human Resource Development Quarterly, 22(2), 157–175. http://dx.doi.org/10.1002/hrdq.20072

DeRue, D. S., & Wellman, N. (2009). Developing leaders via experience: The role of developmental challenge, learning orientation, and feedback availability. Journal of Applied Psychology, 94(4), 859–875. http://dx.doi.org/10.1037/a0015317

Dierdorff, E. C., & Ellington, J. K. (2012). Members matter in team training: Multilevel and longitudinal relationships between goal orientation, self-regulation, and team outcomes. Personnel Psychology, 65(3), 661–703. http://dx.doi.org/10.1111/j.1744-6570.2012.01255.x

Dolmans, D. H. J. M., & Schmidt, H. G. (2006). What do we know about cognitive and motivational effects of small group tutorials in problem-based learning? Advances in Health Sciences Education, 11(4), 321–336. http://dx.doi.org/10.1007/s10459-006-9012-8

Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41(10), 1040–1048. http://dx.doi.org/10.1037/0003-066X.41.10.1040

Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95(2), 256–273. http://dx.doi.org/10.1037/0033-295X.95.2.256

Garavan, T. N., McGuire, D., & O’Donnell, D. (2004). Exploring human resource development: A levels of analysis approach. Human Resource Development Review, 3(4), 417–441. http://dx.doi.org/10.1177/1534484304271669

Gijselaers, W. H., & Schmidt, H. G. (1990). Towards a causal model of student learning within the context of a problem- based curriculum. In Z. M. Nooman, H. G. Schmidt, & E. S. Ezzat (Eds.), Innovation in medical education: An evaluation of its present status (pp. 95–114). New York, NY: Springer.

Goltz, S. M., Hietapelto, A. B., Reinsch, R. W., & Tyrell, S. K. (2008). Teaching teamwork and problem solving concurrently. Journal of Management Education, 32(5), 541–562. http://dx.doi.org/10.1177/1052562907310739

Hallinger, P., & Lu, J. (2011). Assessing the instructional effectiveness of problem-based management education in Thailand: A longitudinal evaluation. Management Learning, 42(3), 279–299. http://dx.doi.org/10.1177/1350507610388596

Harman, H. H. (1976). Modern factor analysis (3rd ed.). Chicago, IL: University of Chicago Press.

Hmelo, C. E., & Lin, X. (2000). Becoming self-directed learners: Strategy development in problem-based learning. In D. H. Evensen & C. E. Hmelo (Eds.), Problem-based learning: A research perspective on learning interactions (pp. 227–250). Mahwah, NJ: Lawrence Erlbaum Associates.

Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266. http://dx.doi.org/10.1023/B:EDPR.0000034022.16470.f3

Hmelo-Silver, C. E., & Barrows, H. S. (2008). Facilitating collaborative knowledge building. Cognition and Instruction, 26(1), 48–94. http://dx.doi.org/10.1080/07370000701798495

Hmelo-Silver, C. E., Duncan, R. G., & Chinn, C. A. (2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 99–107. http://dx.doi.org/10.1080/00461520701263368

Hofmann, D. A. (1997). An overview of the logic and rationale of hierarchical linear models. Journal of Management, 23(6), 723–744. http://dx.doi.org/10.1177/014920639702300602

House, R., Rousseau, D. M., & Thomas-Hunt, M. (1995). The meso paradigm: A framework for the integration of micro and macro organizational behavior. Research in Organizational Behavior, 17, 71–115.

Hung, W. (2006). The 3C3R model: A conceptual framework for designing problems in PBL. Interdisciplinary Journal of Problem-based Learning, 1(1), 55–77. http://dx.doi.org/10.7771/1541-5015.1006

Hung, W. (2009). The 9-step problem design process for problem-based learning: Application of the 3C3R model. Educational Research Review, 4(2), 118–141. http://dx.doi.org/10.1016/j.edurev.2008.12.001

Hung, W. (2011). Theory to reality: A few issues in implementing problem-based learning. Educational Technology Research and Development, 59(4), 529–552. http://dx.doi.org/10.1007/s11423-011-9198-1

Hung, W., Jonassen, D. H., & Liu, R. (2007). Problem-based learning. In J. M. Spector, M. D. Merrill, J. van Merrien- boer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (Vol. 1) (pp. 485–506). New York, NY: Lawrence Erlbaum Associates.

Innes, R. B. (2006). What can learning science contribute to our understanding of the effectiveness of problem-based learning groups? Journal of Management Education, 30(6), 751–764. http://dx.doi.org/10.1177/1052562906287966

James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69(1), 85–98. http://dx.doi.org/10.1037/0021-9010.69.1.85

James, L. R., Demaree, R. G., & Wolf, G. (1993). rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78(2), 306–309. http://dx.doi.org/10.1037/0021-9010.78.2.306

Kek, M. Y. C. A., & Huijser, H. (2011). The power of problem- based learning in developing critical thinking skills: Preparing students for tomorrow’s digital futures in today’s classrooms. Higher Education Research and Development, 30(3), 329–341. http://dx.doi.org/10.1080/07294360.2010.501074

Kolb, A. Y., & Kolb, D. A. (2005). Learning styles and learning spaces: Enhancing experiential learning in higher education. Academy of Management Learning and Education, 4(2), 193–212. http://dx.doi.org/10.5465/AMLE.2005.17268566

Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23(2), 127–135. http://dx.doi.org/10.1177/01466219922031257

Liu, D., & Fu, P.-P. (2011). Motivating protégés’ personal learning in teams: A multilevel investigation of autonomy support and autonomy orientation. Journal of Applied Psychology, 96(6), 1195–1208. http://dx.doi.org/10.1037/a0024716

Loyens, S. M. M., Kirschner, P. A., & Paas, F. (2012). Problem-based learning. In K. R. Harris, S. Graham, T. Urdan, A. G. Bus, S. Major, & H. L. Swanson (Eds.), APA educational psychology handbook (Vol.3): Application to learning and teaching (pp. 403–425). Washington, D.C.: American Psychological Association. http://dx.doi.org/10.1037/13275-016

Loyens, S. M. M., Magda, J., & Rikers, R. M. J. P. (2008). Self-directed learning in problem-based learning and its relationships with self-regulated learning. Educational Psychology Review, 20(4), 411–427. http://dx.doi.org/10.1007/s10648-008-9082-7

Mamede, S., Schmidt, H. G., & Norman, G. R. (2006). Innovations in problem-based learning: What can we learn from recent studies? Advances in Health Sciences Education, 11(4), 403–422. http://dx.doi.org/10.1007/s10459-006-9018-2

Marquardt, M. J. (1999). Action learning in action: Transforming problems and people for world-class organizational learning. Palo Alto, CA: Davies-Black Publishing.

Miflin, B. M., Campbell, C. B., & Price, D. A. (2000). A conceptual framework to guide the development of self-directed, lifelong learning in problem-based medical curricula. Medical Education, 34(4), 299–306. http://dx.doi.org/10.1046/j.1365-2923.2000.00564.x

Molleman, E., Nauta, A., & Buunk, B. P. (2007). Social comparison-based thoughts in groups: Their associations with interpersonal trust and learning outcomes. Journal of Applied Social Psychology, 37(6), 1163–1180. http://dx.doi.org/10.1111/j.1559-1816.2007.00207.x

Neufeld, V. R., & Barrows, H. S. (1974). The “McMaster Philosophy”: An approach to medical education. Journal of Medical Education, 49(11), 1040–1050.

Newman, M. (2006). Fitness for purpose evaluation in problem based learning should consider the requirements for establishing descriptive causation. Advances in Health Sciences Education, 11(4), 391–402. http://dx.doi.org/10.1007/s10459-006-9017-3

Pease, M. A., & Kuhn, D. (2011). Experimental analysis of the effective components of problem-based learning. Science Education, 95(1), 57–86. http://dx.doi.org/10.1002/sce.20412

Phan, H. P. (2009). Exploring students’ reflective thinking practice, deep processing strategies, effort, and achievement goal orientations. Educational Psychology, 29(3), 297–313. http://dx.doi.org/10.1080/01443410902877988

Pieterse, A. N., van Knippenberg, D., & van Ginkel, W. P. (2011). Diversity in goal orientation, team reflexivity, and team performance. Organizational Behavior and Human Decision Processes, 114(2), 153–164. http://dx.doi.org/10.1016/j.obhdp.2010.11.003

Poortvliet, P. M., Janssen, O., van Yperen, N. W., & van de Vliert, E. (2007). Achievement goals and interpersonal behavior: How mastery and performance goals shape information exchange. Personality and Social Psychology Bulletin, 33(10), 1435–1447. http://dx.doi.org/10.1177/0146167207305536

Raelin, J. A. (1997). Individual and situational precursors of successful action learning. Journal of Management Education, 21(3), 368–394. http://dx.doi.org/10.1177/105256299702100308

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications, Inc.

Reiser, B. J. (2004). Scaffolding complex learning: The mechanisms of structuring and problematizing student work. Journal of the Learning Sciences, 13(3), 273–304. http://dx.doi.org/10.1207/s15327809jls1303_2

Rotgans, J. I., & Schmidt, H. G. (2011). Cognitive engagement in the problem-based learning classroom. Advances in Health Sciences Education, 16(4), 465–479. http://dx.doi.org/10.1007/s10459-011-9272-9

Schmidt, H. G. (1983). Problem-based learning: Rationale and description. Medical Education, 17(1), 11–16. http://dx.doi.org/10.1111/j.1365-2923.1983.tb01086.x

Schmidt, H. G., & Moust, J. H. C. (2000). Factors affecting small-group learning: A review of the research. In D. H. Evensen & C. E. Hmelo (Eds.), Problem-based learning: A research perspective on learning interactions (pp. 19–52). Mahwah, NJ: Lawrence Erlbaum.

Schmidt, H. G., Rotgans, J. I., & Yew, E. H. J. (2011). The process of problem-based learning: What works and why. Medical Education, 45(8), 792–806. http://dx.doi.org/10.1111/j.1365-2923.2011.04035.x

Schmidt, H. G., van der Molen, H. T., te Winkel, W. W. R., & Wijnen, W. H. F. W. (2009). Constructivist, problem-based learning does work: A meta-analysis of curricular comparisons involving a single medical school. Educational Psychologist, 44(4), 227–249. http://dx.doi.org/10.1080/00461520903213592

Seibert, S. E., Silver, S. R., & Randolph, W. A. (2004). Taking empowerment to the next level: A multiple-level model of empowerment, performance and, satisfaction. Academy of Management Journal, 47(3), 332–349. http://dx.doi.org/10.2307/20159585

Senko, C., Hulleman, C. S., & Harackiewicz, J. M. (2011). Achievement goal theory at the crossroads: Old controversies, current challenges, and new directions. Educational Psychologist, 46(1), 26–47. http://dx.doi.org/10.1080/00461520.2011.538646

Sherwood, A. L. (2004). Problem-based learning in management education: A framework for designing context. Journal of Management Education, 28(5), 536–557. http://dx.doi.org/10.1177/1052562904265773

Sitzmann, T., Ely, K., Brown, K. G., & Bauer, K. N. (2010). Self-assessment of knowledge: A cognitive learning or affective measure? Academy of Management Learning and Education, 9(2), 169–191. http://dx.doi.org/10.5465/AMLE.2010.51428542

Smith, G. F. (2005). Problem-based learning: Can it improve managerial thinking? Journal of Management Education, 29(2), 357–378. http://dx.doi.org/10.1177/1052562904269642

Sockalingam, N., Rotgans, J. I., & Schmidt, H. G. (2011). The relationships between problem characteristics, achievement-related behaviors, and academic achievement in problem-based learning. Advances in Health Sciences Education, 16(4), 481–490. http://dx.doi.org/10.1007/s10459-010-9270-3

Sockalingam, N., & Schmidt, H. G. (2011). Characteristics of problems for problem-based learning: The students’ perspective. Interdisciplinary Journal of Problem-based Learning, 5(1), 6–33. http://dx.doi.org/10.7771/1541-5015.1135

Svinicki, M. D. (2007). Moving beyond “it worked”: The ongoing evolution of research on problem-based learning in medical education. Educational Psychology Review, 19(1), 49–61. http://dx.doi.org/10.1007/s10648-006-9040-1

van Berkel, H. J. M., & Schmidt, H. G. (2000). Motivation to commit oneself as a determinant of achievement in problem-based learning. Higher Education, 40(2), 231–242. http://dx.doi.org/10.1023/A:1004022116365

van den Hurk, M. M., Dolmans, D. H. J. M, Wolfhagen, I. H. A. P., & van der Vleuten, C. P. M. (2001). Testing a causal model for learning in a problem-based curriculum. Advances in Health Sciences Education, 6(2), 141–149. http://dx.doi.org/10.1023/A:1011402507476

van Knippenberg, D., & Schippers, M. C. (2007). Work group diversity. Annual Review of Psychology, 58(1), 515– 541. http://dx.doi.org/10.1146/annurev.psych.58.110405.085546

Walker, A., & Leary, H. (2009). A problem based learning meta analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem-based Learning, 3(1), 12–43. http://dx.doi.org/10.7771/1541-5015.1061

Wexley, K. N., & Baldwin, T. T. (1986). Posttraining strategies for facilitating positive transfer: An empirical exploration. Academy of Management Journal, 29(3), 503–520. http://dx.doi.org/10.2307/256221

Wirkala, C., & Kuhn, D. (2011). Problem-based learning in K–12 education: Is it effective and how does it achieve its effects? American Educational Research Journal, 48(5), 1157–1186. http://dx.doi.org/10.3102/0002831211419491