Using Analytics to Transform a Problem-Based Case Library: An Educational Design Research Approach
Main Article Content
Abstract
This article describes the iterative design, development, and evaluation of a case-based learning environment focusing on an ill-structured sales management problem. We discuss our processes and situate them within the broader framework of educational design research. The learning environment evolved over the course of three design phases. A semisummative evaluation of student concept maps after the third phase revealed unsatisfactory learning outcomes. This paper focuses on how we investigated design flaws that contributed to poor learning performance. A specific focus of our investigation was the use of Google Analytics data, which uncovered weaknesses in our design. Based on our findings, we used a rapid prototyping process to redesign the learning environment, emphasizing interactive and multimedia-rich elements. Processes and methods are reported along with discussion of implications for case-based reasoning, including relevant design principles. This article will provide insights into resolving design tensions for researchers and practitioners seeking to advance theory and practice in similar domains.
Article Details
1. Publication and Promotion: In consideration of the Publisher’s agreement to publish the Work, Author hereby grants and assigns to Publisher the non-exclusive right to print, publish, reproduce, or distribute the Work throughout the world in all means of expression by any method now known or hereafter developed, including electronic format, and to market or sell the Work orany part of it as Publisher sees fit. Author further grants Publisher the right to use Author’s name in association with the Work inpublished form and in advertising and promotional materials
2. Copyright: Copyright of the Work remains in Author’s name.
3. Prior Publication and Attribution: Author agrees not to publish the Work in print form prior to publication of the Work by the Publisher. Author agrees to cite, by author, title, and publisher, the original Interdisciplinary Journal of Problem-based Learning publication when publishing the Work elsewhere
4. Author Representations: The Author represents and warrants that the Work:
(a) is the Author’s original Work and that Author has full power to enter into this Agreement;
(b) does not infringe the copyright or property of another;
(c) contains no material which is obscene, libelous, defamatory or previously published, in whole or in part.
Author shall indemnify and hold Publisher harmless against loss of expenses arising from breach of any such warranties.
5. Licensing and Reuse: Reuse of the published Work will be governed by a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0; http://creativecommons.org/licenses/ by-nc/4.0/). This license lets others remix, tweak, and build upon the Work non-commercially; although new works must acknowledge the original Interdisciplinary Journal of Problem-based Learning publication and be non-commercial, they do not have to be licensed on the same terms.
References
Aha, D. W., McSherry, D., & Yang, Q. (2005). Advances in conversational case-based reasoning. Knowledge Engineering Review, 20(3), 247–254.
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational researcher, 41(1), 16–25. https://doi.org/10.3102/0013189X11428813
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sci- ences, 13(1), 1–14.
Bennett, S. (2010). Investigating strategies for using related cases to support design problem solving. Educational Technology Research and Development, 58(4), 459–480.
Branting, K., Lester, J., & Mott, B. (2004). Dialogue management for conversational case-based reasoning. In P. Funk & P. A. González Calero (Eds.), Advances in Case-Based Reasoning. ECCBR 2004. Lecture Notes in Computer Science, Vol. 3155 (pp. 77–90). Berlin, Heidelberg: Springer.
Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.
Curwood, J. S., Tomitsch, M., Thomson, K., & Hendry, G. D. (2015). Professional learning in higher education: Understanding how academics interpret student feedback and access resources to improve their teaching. Australasian Journal of Educational Technology, 31(5), 556–571.
Ertmer, P. A., Schlosser, S., Clase, K., & Adedokun, O. (2014). The grand challenge: Helping teachers learn/teach cutting-edge science via a PBL approach. Interdisciplinary Journal of Problem-Based Learning, 8(1), 1–20.
Ertmer, P., & Koehler, A. A. (2014). Online case-based dis- cussions: Examining coverage of the afforded problem space. Educational Technology Research & Development, 62(5), 617–636.
Ferguson, W., Bareiss, R., Birnbaum, L., & Osgood, R. (1992). ASK Systems: An approach to the realization of story-based teachers. Journal of the Learning Sciences, 2(1), 95–134.
Fitzgerald, G., Koury, K., Mitchem, K., Hollingsead, C., Miller, K., Park, M. K., & Tsai, H.-H. (2009). Implementing case-based instruction in higher education through technology: What works best? Journal of Technology and Teacher Education, 17(1), 31–63.
Fitzgerald, G., Mitchem, K., Hollingsead, C., Miller, K., Koury, K., & Tsai, H.-H. (2011). Exploring the bridge from multimedia cases to classrooms: Evidence of transfer. Journal of Special Education Technology, 26(2), 23–38.
Gartmeier, M., Bauer, J., Fischer, M. R., Hoppe-Seyler, T., Karsten, G., Kiessling, C., . . . Prenzel, M. (2015). Fostering professional communication skills of future physicians and teachers: Effects of e-learning with video cases and roleplay. Instructional Science, 43(4), 443–462.
Ge, X., & Land, S. (2003). Scaffolding students’ problem solving processes in an ill-structured task using question prompts and peer interactions. Educational Technology Research & Development, 51(1), 21–38.
Harackiewicz, J. M., Barron, K. E., Tauer, J. M., Carter, S. M., & Elliot, A. J. (2000). Short-term and long-term consequences of achievement goals: Predicting interest and performance over time. Journal of Educational Psychology, 92(2), 316–330.
Henry, H., Tawfik, A. A., Jonassen, D. H., Winholtz, R., & Khanna, S. (2012). “I know this is supposed to be more like the real world, but . . .”: Student perceptions of a PBL implementation in an undergraduate materials science course. Interdisciplinary Journal of Problem-Based Learning, 6(1). https://doi.org/10.7771/1541-5015.1312
Herrington, J., Reeves, T. C., & Oliver, R. (2014). Authentic learning environments. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 453–464). New York: Springer.
Hmelo-Silver, C., Derry, S. J., Bitterman, A., & Hatrak, N. (2009). Targeting transfer in a STELLAR PBL course for pre-service teachers. Interdisciplinary Journal of Problem- Based Learning, 3(2), 24–42.
Jonassen, D. H. (2011). ASK Systems: Interrogative access to multiple ways of thinking. Educational Technology Research and Development, 59(1), 159–175.
Jonassen, D. H. (2011). Learning to solve problems: A handbook for designing problem solving learning environments (1st ed.). Routledge.
Jonassen, D. H., & Cho, Y. (2011). Fostering argumentation while solving engineering ethics problems. Journal of Engineering Education, 100(4), 680–702.
Jonassen, D. H., & Hernandez-Serrano, J. (2002). Case-based reasoning and instructional design: Using stories to support problem solving. Educational Technology, Research, & Development, 50(2), 65–77.
Kim, H., & Hannafin, M. J. (2011). Developing situated knowledge about teaching with technology via web-enhanced case-based activity. Computers & Education, 57(1), 1378–1388.
Kirschner, P., & van Merriënboer, J. J. G. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183.
Kirschner, P., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86.
Kolodner, J., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., . . . Ryan, M. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design into practice. Journal of the Learning Sciences, 12(4), 495–547.
Kolodner, J., Dorn, B., Owensby, J., & Guzdial, M. (2012). Theory and practice of case-based learning aids. In D. H. Jonassen & S. Land (Eds.), Theoretical foundations of learning environments (2nd ed., pp. 142–170). New York: Routledge.
Kolodner, J., Owensby, J., & Guzdial, M. (2004). Case-based learning aids. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology: A project of the Association for Educational Communications and Technology (2nd ed., pp. 829–861). Mahwah, NJ: Lawrence Earlbaum Associates.
Kopcha, T. J., McGregor, J., Shin, S., Qian, Y., Choi, J., Hill, R., . . . & Choi, I. (2017). Developing an integrative STEM curriculum for robotics education through educational design research. Journal of Formative Design in Learning, 1(1), 1–14.
Kopcha, T. J., Schmidt, M., & McKenney, S. (2015). Edito- rial 31(5): Preface to the special issue. In T. J. Kopcha, M. Schmidt, & S. McKenney (Eds.), Educational design research in post-secondary learning environments. Australasian Journal of Educational Technology, 31(5), i–ix.
Lazonder, A. (2014). Inquiry learning. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 453–464). New York: Springer.
Lazonder, A., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: Effects of guidance. Review of Educational Research, 87(4), 1–38.
Leary, H., & Walker, A. (2009). A problem-based learning meta analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem-Based Learning, 3(1). Retrieved from http://docs.lib.purdue.edu/ijpbl/vol3/iss1/3
Lin-Siegler, X., Shaenfield, D., & Elder, A. D. (2015). Contrasting case instruction can improve self-assessment of writing. Educational Technology Research & Development, 63(4), 517–537.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52.
McKenney, S., & Reeves, T. (2014). Educational design research. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 131–140). New York: Springer.
McSherry, D. (2001). Minimizing dialog length in interactive case-based reasoning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence—Volume 2 (pp. 993–998). San Francisco: Morgan Kaufmann Publishers.
Nielsen, J. (2005). Lower-literacy users: Writing for a broad consumer audience. NN/g. March 14. Retrieved from https://www.nngroup.com/articles/writing-for-lower-literacy-users/
Nielsen, J. (2006). F-shaped pattern for reading web content. NN/g. April 17. Retrieved from https://www.nngroup.com/articles/f-shaped-pattern-reading-web-content/
Noyes, J. M., & Garland, K. J. (2008). Computer- vs. paper- based tasks: Are they equivalent? Ergonomics, 51(9), 1352–1375.
Phillips, D. C., & Dolle, J. R. (2006). From Plato to Brown and beyond: Theory, practice, and the promise of design experiments. In L. Verschaffel, F. Dochy, M. Boekaerts, & S. Vosniadou (Eds.), Instructional psychology: Past, present and future trends: Sixteen essays in honour of Erik DeCorte (pp. 277–293). Amsterdam: Elsevier.
Plomp, T. (2013). Educational design research: An introduc- tion. In T. Plomp & N. Nieveen (Eds.), Educational design research: An introduction (pp. 10–51). Enschede: Netherlands Institute for Curriculum Development.
Reeves, T., Herrington, J., & Oliver, R. (2005). Design research: A socially responsible approach to instructional technology research in higher education. Journal of Computing in Higher Education, 16(2), 96–115.
Schank, R. (1999). Dynamic memory revisited (2nd ed.). Cambridge: Cambridge University Press.
Tawfik, A. A., & Jonassen, D. H. (2013). The effects of successful versus failure-based cases on argumentation while solving decision-making problems. Educational Technology Research & Development, 61(3), 385–406.
Tawfik, A. A., & Kolodner, J. (2016). Systematizing scaffolding for problem-based learning: A view from case-based reasoning. Interdisciplinary Journal of Problem-Based Learning, 10(1).
Tawfik, A. A. (2017). Do cases teach themselves? A comparison of case library prompts in supporting problem- solving during argumentation. Journal of Computing in Higher Education, 29(2), 267–285.
Tripp, S. D., & Bichelmeyer, B. (1990). Rapid prototyping: An alternative instructional design strategy. Educational Technology Research and Development, 38(1), 31–44.
van Merriënboer, J. J. G. (2013). Perspectives on problem solving and instruction. Computers & Education, 64, 153–160.
Wozniak, H. (2015). Conjecture mapping to optimize the educational design research process. Australasian Journal of Educational Technology, 31(5), 597–612.
Zuiker, S. J., & Wright, K. (2015). Learning in and beyond school gardens with cyber-physical systems. Interactive Learning Environments, 23(5), 556–577.