Using Analytics to Transform a Problem-Based Case Library: An Educational Design Research Approach

Main Article Content

Matthew Schmidt
Andrew A. Tawfik


This article describes the iterative design, development, and evaluation of a case-based learning environment focusing on an ill-structured sales management problem. We discuss our processes and situate them within the broader framework of educational design research. The learning environment evolved over the course of three design phases. A semisummative evaluation of student concept maps after the third phase revealed unsatisfactory learning outcomes. This paper focuses on how we investigated design flaws that contributed to poor learning performance. A specific focus of our investigation was the use of Google Analytics data, which uncovered weaknesses in our design. Based on our findings, we used a rapid prototyping process to redesign the learning environment, emphasizing interactive and multimedia-rich elements. Processes and methods are reported along with discussion of implications for case-based reasoning, including relevant design principles. This article will provide insights into resolving design tensions for researchers and practitioners seeking to advance theory and practice in similar domains.

Article Details



Aha, D. W., McSherry, D., & Yang, Q. (2005). Advances in conversational case-based reasoning. Knowledge Engineering Review, 20(3), 247–254.

Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational researcher, 41(1), 16–25.

Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sci- ences, 13(1), 1–14.

Bennett, S. (2010). Investigating strategies for using related cases to support design problem solving. Educational Technology Research and Development, 58(4), 459–480.

Branting, K., Lester, J., & Mott, B. (2004). Dialogue management for conversational case-based reasoning. In P. Funk & P. A. González Calero (Eds.), Advances in Case-Based Reasoning. ECCBR 2004. Lecture Notes in Computer Science, Vol. 3155 (pp. 77–90). Berlin, Heidelberg: Springer.

Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.

Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.

Curwood, J. S., Tomitsch, M., Thomson, K., & Hendry, G. D. (2015). Professional learning in higher education: Understanding how academics interpret student feedback and access resources to improve their teaching. Australasian Journal of Educational Technology, 31(5), 556–571.

Ertmer, P. A., Schlosser, S., Clase, K., & Adedokun, O. (2014). The grand challenge: Helping teachers learn/teach cutting-edge science via a PBL approach. Interdisciplinary Journal of Problem-Based Learning, 8(1), 1–20.

Ertmer, P., & Koehler, A. A. (2014). Online case-based dis- cussions: Examining coverage of the afforded problem space. Educational Technology Research & Development, 62(5), 617–636.

Ferguson, W., Bareiss, R., Birnbaum, L., & Osgood, R. (1992). ASK Systems: An approach to the realization of story-based teachers. Journal of the Learning Sciences, 2(1), 95–134.

Fitzgerald, G., Koury, K., Mitchem, K., Hollingsead, C., Miller, K., Park, M. K., & Tsai, H.-H. (2009). Implementing case-based instruction in higher education through technology: What works best? Journal of Technology and Teacher Education, 17(1), 31–63.

Fitzgerald, G., Mitchem, K., Hollingsead, C., Miller, K., Koury, K., & Tsai, H.-H. (2011). Exploring the bridge from multimedia cases to classrooms: Evidence of transfer. Journal of Special Education Technology, 26(2), 23–38.

Gartmeier, M., Bauer, J., Fischer, M. R., Hoppe-Seyler, T., Karsten, G., Kiessling, C., . . . Prenzel, M. (2015). Fostering professional communication skills of future physicians and teachers: Effects of e-learning with video cases and roleplay. Instructional Science, 43(4), 443–462.

Ge, X., & Land, S. (2003). Scaffolding students’ problem solving processes in an ill-structured task using question prompts and peer interactions. Educational Technology Research & Development, 51(1), 21–38.

Harackiewicz, J. M., Barron, K. E., Tauer, J. M., Carter, S. M., & Elliot, A. J. (2000). Short-term and long-term consequences of achievement goals: Predicting interest and performance over time. Journal of Educational Psychology, 92(2), 316–330.

Henry, H., Tawfik, A. A., Jonassen, D. H., Winholtz, R., & Khanna, S. (2012). “I know this is supposed to be more like the real world, but . . .”: Student perceptions of a PBL implementation in an undergraduate materials science course. Interdisciplinary Journal of Problem-Based Learning, 6(1).

Herrington, J., Reeves, T. C., & Oliver, R. (2014). Authentic learning environments. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 453–464). New York: Springer.

Hmelo-Silver, C., Derry, S. J., Bitterman, A., & Hatrak, N. (2009). Targeting transfer in a STELLAR PBL course for pre-service teachers. Interdisciplinary Journal of Problem- Based Learning, 3(2), 24–42.

Jonassen, D. H. (2011). ASK Systems: Interrogative access to multiple ways of thinking. Educational Technology Research and Development, 59(1), 159–175.

Jonassen, D. H. (2011). Learning to solve problems: A handbook for designing problem solving learning environments (1st ed.). Routledge.

Jonassen, D. H., & Cho, Y. (2011). Fostering argumentation while solving engineering ethics problems. Journal of Engineering Education, 100(4), 680–702.

Jonassen, D. H., & Hernandez-Serrano, J. (2002). Case-based reasoning and instructional design: Using stories to support problem solving. Educational Technology, Research, & Development, 50(2), 65–77.

Kim, H., & Hannafin, M. J. (2011). Developing situated knowledge about teaching with technology via web-enhanced case-based activity. Computers & Education, 57(1), 1378–1388.

Kirschner, P., & van Merriënboer, J. J. G. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183.

Kirschner, P., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86.

Kolodner, J., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., . . . Ryan, M. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design into practice. Journal of the Learning Sciences, 12(4), 495–547.

Kolodner, J., Dorn, B., Owensby, J., & Guzdial, M. (2012). Theory and practice of case-based learning aids. In D. H. Jonassen & S. Land (Eds.), Theoretical foundations of learning environments (2nd ed., pp. 142–170). New York: Routledge.

Kolodner, J., Owensby, J., & Guzdial, M. (2004). Case-based learning aids. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology: A project of the Association for Educational Communications and Technology (2nd ed., pp. 829–861). Mahwah, NJ: Lawrence Earlbaum Associates.

Kopcha, T. J., McGregor, J., Shin, S., Qian, Y., Choi, J., Hill, R., . . . & Choi, I. (2017). Developing an integrative STEM curriculum for robotics education through educational design research. Journal of Formative Design in Learning, 1(1), 1–14.

Kopcha, T. J., Schmidt, M., & McKenney, S. (2015). Edito- rial 31(5): Preface to the special issue. In T. J. Kopcha, M. Schmidt, & S. McKenney (Eds.), Educational design research in post-secondary learning environments. Australasian Journal of Educational Technology, 31(5), i–ix.

Lazonder, A. (2014). Inquiry learning. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 453–464). New York: Springer.

Lazonder, A., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: Effects of guidance. Review of Educational Research, 87(4), 1–38.

Leary, H., & Walker, A. (2009). A problem-based learning meta analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem-Based Learning, 3(1). Retrieved from

Lin-Siegler, X., Shaenfield, D., & Elder, A. D. (2015). Contrasting case instruction can improve self-assessment of writing. Educational Technology Research & Development, 63(4), 517–537.

Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52.

McKenney, S., & Reeves, T. (2014). Educational design research. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 131–140). New York: Springer.

McSherry, D. (2001). Minimizing dialog length in interactive case-based reasoning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence—Volume 2 (pp. 993–998). San Francisco: Morgan Kaufmann Publishers.

Nielsen, J. (2005). Lower-literacy users: Writing for a broad consumer audience. NN/g. March 14. Retrieved from

Nielsen, J. (2006). F-shaped pattern for reading web content. NN/g. April 17. Retrieved from

Noyes, J. M., & Garland, K. J. (2008). Computer- vs. paper- based tasks: Are they equivalent? Ergonomics, 51(9), 1352–1375.

Phillips, D. C., & Dolle, J. R. (2006). From Plato to Brown and beyond: Theory, practice, and the promise of design experiments. In L. Verschaffel, F. Dochy, M. Boekaerts, & S. Vosniadou (Eds.), Instructional psychology: Past, present and future trends: Sixteen essays in honour of Erik DeCorte (pp. 277–293). Amsterdam: Elsevier.

Plomp, T. (2013). Educational design research: An introduc- tion. In T. Plomp & N. Nieveen (Eds.), Educational design research: An introduction (pp. 10–51). Enschede: Netherlands Institute for Curriculum Development.

Reeves, T., Herrington, J., & Oliver, R. (2005). Design research: A socially responsible approach to instructional technology research in higher education. Journal of Computing in Higher Education, 16(2), 96–115.

Schank, R. (1999). Dynamic memory revisited (2nd ed.). Cambridge: Cambridge University Press.

Tawfik, A. A., & Jonassen, D. H. (2013). The effects of successful versus failure-based cases on argumentation while solving decision-making problems. Educational Technology Research & Development, 61(3), 385–406.

Tawfik, A. A., & Kolodner, J. (2016). Systematizing scaffolding for problem-based learning: A view from case-based reasoning. Interdisciplinary Journal of Problem-Based Learning, 10(1).

Tawfik, A. A. (2017). Do cases teach themselves? A comparison of case library prompts in supporting problem- solving during argumentation. Journal of Computing in Higher Education, 29(2), 267–285.

Tripp, S. D., & Bichelmeyer, B. (1990). Rapid prototyping: An alternative instructional design strategy. Educational Technology Research and Development, 38(1), 31–44.

van Merriënboer, J. J. G. (2013). Perspectives on problem solving and instruction. Computers & Education, 64, 153–160.

Wozniak, H. (2015). Conjecture mapping to optimize the educational design research process. Australasian Journal of Educational Technology, 31(5), 597–612.

Zuiker, S. J., & Wright, K. (2015). Learning in and beyond school gardens with cyber-physical systems. Interactive Learning Environments, 23(5), 556–577.