The impact of integrated student experiences on learning

In a cluster of courses called Food for Thought, seven faculty from different departments (including Biology, Economics, Sociology, Chemistry, Health and Wellness, and Foreign Language) teach students about food information, food consumerism, nutrition and health. The classes all have a shared learning outcome focused on developing the student as an informed consumer of food. Each semester, faculty teach a food-related course from their respective disciplinary perspective while also incorporating cross-course interactions that allow for both integration of disciplinary knowledge and studentto-student learning opportunities. Previous research demonstrated that this approach leads to student perceptions of learning gains (Wingert et al., 2011). Building on that work, this research directly assesses student learning in the Food for Thought cluster. The results demonstrate that exposure to multiple disciplines covering a shared topic enhances learning through greater student ability to integrate diverse forms of knowledge and to see an issue from multiple perspectives. These findings demonstrate the value of multidisciplinary learning opportunities for students.


Background
Previous research analyzing the effects of integrative learning often focuses on learning communities, or explicit attempts to use a cross-course structure to teach across disciplines.This research is useful in illuminating a number of the benefits of an interdisciplinary approach to teaching and learning.Studies using a variety of largely qualitative methods also report benefits of intentional integrative learning environments.Gray (2000) summarizes assessment results from a number of colleges and universities that document improvement in various indicators of success including retention, progress towards degree completion, measures of personal and intellectual satisfaction, and comparisons of student surveys with those from classes not part of a learning community.Tinto (2000) similarly reports increased retention among students in learning communities as well as an increased sense of responsibility on the part of the students for their learning as well as that of the other students.In a comparison of students enrolled and not enrolled in a learning community, students in a learning community reported an enhanced ability to transfer knowledge from one course to another (Inkelas, Vogt, Longerbeam, & Johnson, 2006).Stefanou & Salisbury-Glennon (2001) document improved motivation and cognitive development in learning communities and Avens & Zelley (1992) found improvement in an instrument called a "Measure of Intellectual Development."MacGregor, Tinto, and Lindblad (2000) reviewed seventy assessment studies of learning communities and determined that they facilitate interdisciplinary connections and more profound understandings of content.Stefanou & Salisbury-Glennon (2001) document improved motivation and cognitive development in learning communities.
In more recent reports survey data with Likert scores or student course evaluations have been used to measure the effects of intentional integrative learning strategies (Wingert et al., 2011, Cohen, 2010;Escribano, Agüera, and Tovar 2011;Mahoney & Schamber, 2011;and Melendez, Bowman, Erickson, and Swim, 2009).Although these studies present findings without a comparison control group, they describe a number of important innovations.For instance, Escribano et al. (2011) tasked physiology students with producing, as a group, a newsletter that compelled them to apply a number of new tools.The authors scored student work on a Likert scale and reported positive assessments, based on measures of writing, critical thinking, and oral skills.Mahoney & Schamber (2011) describe a pairing of a general education seminar course with a public speaking course to foster critical thinking and oral skills.They report finding an unusual level of sophistication in first year students and credit explicit connections between the two courses for this effect.Cohen (2010) joined students from two major divisions of the school (the design school and the liberal arts school) and engaged them in cross-divisional teaching in a new course that used student projects focused on urban food and agricultural systems.Mid-and end-semester survey data included mostly positive student comments with several complaints about group work.Melendez et al. (2009) reported combining math, science, and engineering majors with humanities majors in a seminar.Students were tasked with designing mathematical models to describe biological, economic, and sociological phenomena and were supervised by faculty in these departments.The authors found more positive responses on end of course evaluations than were found the previous year before implementation of the multidisciplinary activities.
Using a number of qualitative methods, Mulligan, Taylor, Glen, Tomlin, and Gaul (2011) described a cross-disciplinary experience designed for chemistry and kinesiology majors.Exercise physiology and analytical chemistry students conducted electrolyte analysis of sweat from participants during exercise.Focus groups were used to gather data from the students.Transcripts from focus groups were coded based on topics that emerged as common themes among the focus groups.Students had positive comments about their interest and enthusiasm, the value of student interactions, the influence of this exercise on the course content, and applicability of this exercise to the real world.Students had negative feedback about the extra work required.
In previous work from this team of authors, using a 61-item entrance and exit survey for each class experience (n =106) in the Food for Thought cluster, strong student perceptions of learning gains were demonstrated.The learning outcomes questions were organized into seven categories: academic attitudes, civic engagement, informed consumer, food literacy, research literacy, information and communication skills, and understanding food systems.As a whole, the Food for Thought cluster had a statistically significant positive impact on student's perception of their learning with the strongest impacts on food literacy, understanding food systems and interest in civic engagement related to food systems.Questions related to food literacy (disciplinary and interdisciplinary skills), informed consumption and understanding food systems are all related to integrated learning.(Wingert et al., 2011) The Intentional Integrative Learning Experiences At UNC Asheville students take their general education distribution in natural science, social science, and humanities or arts in topical clusters centered on a common theme such as poverty and human capability, Black creative experiences, or globalization.Students participate in one of 15 clusters by completing three courses from three different disciplines, of which there is at least one natural science and at least one social science.Students must declare and complete one cluster in order to graduate but students can enroll in individual cluster courses without declaring the cluster and many of the courses in clusters may also fulfill general education or major requirements.This paper focuses on one of those clusters, Food for Thought: Engaging the Citizen in the Science and Politics of Food Information, Food Consumerism, Nutrition and Health (Food for Thought).Table 1 describes the 9 courses in this cluster in Academic Year 2011-2012.
Initiated in 2007, the Food for Thought cluster focuses on developing the student as an informed consumer of food by providing a platform for discussion of what we eat, why we eat, where our food comes from and its journey from production to consumption, and how food affects our bodies and health.As human beings, our bodies and our societies are interlinked by numerous processes, many of which can be understood by investigating the dynamics of food in chemical, biological, cultural and social systems.Our primary goal for students is an enhanced, interdisciplinary understanding of the interplay of these systems and a more attuned sense of how food is a civic issue.Together, the seven faculty teaching in the Food for Thought cluster developed a set of shared learning outcomes that inform not only individual courses in the cluster, but shared learning opportunities among the courses.These outcomes included academic attitudes, civic engagement, informed consumerism, interdisciplinary and disciplinary skills (food and research literacy), information and communication skills, food systems and their interrelatedness with social processes, environmental systems, and individual health.Each semester faculty teaching in the Food for Thought cluster in any term during the year convene regularly to plan and implement an appropriate set of cluster activities for the courses that are being offered that term.Students engage with the cluster themes by participating in course-specific projects and activities, such as measuring the content of calcium and iron in foods and supplements (chemistry) or studying local food distribution systems (economics, health and wellness); crosscourse cluster projects that engage students across cluster courses (such as creating and evaluating an urban community garden or critiquing media messages around weight management and food products) and cross-course cluster activities that are available to one or more courses in a given semester (such as farm tours and seminar series).This paper focuses on the novel dimensions of our cluster of courses in a general education program -those that involve engagement outside of traditional classroom configurations.
Cross-Course Cluster Projects are integrated, multidisciplinary projects that engage students from multiple Food for Thought cluster courses simultaneously.Many involve experiential learning, some engage students in service learning opportunities.See Table 2 for the cross course projects in 2011-12.The students must learn how the knowledge from one discipline relates to others in order to successfully complete the project.Through this process, students in a variety of disciplines in the natural sciences, social sciences and humanities interact with one another and benefit from a range of perspectives for addressing a single issue.
Cross-Course Cluster Activities vary by semester depending on the courses offered and resources available and include attending talks by guest speakers and lunch and learn seminar series, participation in food education activities such as Food Day--a nationwide celebration and a movement for healthy, affordable, and sustainable food events (www.foodday.org),organic farming conferences, touring farms, and working in a campus garden maintained by students or community gardens.For a more in depth explanation of the university context, cluster learning outcomes or cross-course cluster projects see Wingert et al., 2011.

Study Purpose
Previous research demonstrated that the multidisciplinary cluster approach using crosscourse interactions leads to student perceptions of learning gains (Wingert et al., 2011).While this is an important finding, in this paper we move beyond perceptions to assess directly how student learning about food is affected by experiences designed to integrate student learning across disciplines.We hypothesized that the various interactions of the students in the cluster have a positive impact on integrative student learning in the areas of food and research literacy, critical thinking about and understanding of food systems, and becoming an informed consumer of food.The purpose of this paper is to report on our process for and findings on whether participation in the Food for Thought cluster enhances integrative student learning.To do this we asked students to demonstrate their achievement in integrative learning by writing statements based on prompts about a New York Times article that explained the costs and benefits of the popularity of quinoa (Chenopodium quinoa), a nutritious alternative to cereal grains (pseudocereal), which, although endemic to the Andes, has recently become popular in the U.S. due to its nutritional profile, forcing change onto the culture and economy of Bolivia.We tested whether the exposure to a focused, integrative learning environment could result in superior achievement in comparison to a control group of students based on rubric scores derived from written answers to prompts aimed to demonstrate the students' skills in critical thinking and integrative learning.

Methods
To test if integrative experiences impact student learning, an instrument and corresponding evaluation rubric was developed to assess student learning in the areas of food and research literacy, critical thinking about and understanding of food systems, and becoming an informed consumer of food.The instrument and rubric were loosely based on the Critical Thinking Value Rubric created by the Association of American Colleges and Universities (Rhodes, 2010) and on studies in which critical thinking is assessed by asking students to respond to a specific article or reading.Two studies that informed our protocol prompted students to read a designated article or reading and then to evaluate an issue in written form based upon the article or reading; these responses were then evaluated using a rubric designed to assess critical thinking skills (Miller 2004;Connors 2008).The research team vetted a number of news articles related to current food issues and identified an article, "Quinoa's Global Success Creates Quandary at Home" by S. Romero and S. Shahriari (2011) for the study.The article explores the relationship between the rising popularity of the high-protein grain quinoa in the American diet and the problems and opportunities this creates for the production, market economy, and culture of Bolivian growers of quinoa.After reading the article, students were prompted with the following questions: From your perspective, discuss the issue addressed in the article by answering the following questions.1).In one sentence, state the main issue addressed in the article.2).List two sources utilized in the article and evaluate the usefulness of each source to the reader for understanding the issue.3).In one paragraph, discuss the interconnectedness of the issue identified in (1) with three of the following fields: • chemistry and/or biology • agriculture, food and/or nutrition policy • gender/social inequality • marketing/economics 4).How are your choices as a consumer influenced by this issue?Using material from the article and learned in your courses, explain the tradeoffs (pros and cons) that you face in making the decision.Limit your response to one paragraph.
The instrument and rubric were designed to be applied to any article, essay or paper describing a complex food system and requiring the reader to critically evaluate the information to make a decision as an informed consumer.The article on quinoa was specifically selected for this study because it is complex and interdisciplinary in focus, but the topic was not discussed in any of the Food for Thought cluster courses.Furthermore, the article was written for a general audience, and less than three pages in length, allowing the assignment to be completed within a single class period.
The assessment instrument was given to 161 students in 9 Food for Thought Cluster classes (listed in Table 1) and 177 students in 9 control classes [Health Promotion and Wellness (HWP 153, 2 sections), Women's Health (HWP 154, 2 sections), Quantitative Chemistry Laboratory (CHEM 145), Health and Sexuality (HWP 253), Principles of Macroeconomics (ECON 101), and Principles of Microeconomics (ECON 102, 2 sections)].Control courses were selected because they enroll students from a variety of different disciplinary majors and year in college.The students were all asked to participate in an assessment project that may or may not have relevance to the course content.All course instructors implemented the assessment exercise with the following guidelines: • Students had the opportunity to read the article "Quinoa's Global Success Creates Quandary at Home" by S. Romero and S. Shahriari (2011) before class and again during class before answering the exercise questions.• The questions were given out and the assessment exercise was completed during class.
Students had access to the article during the exercise and were given 45 minutes to answer the questions.• So that students took the assessment exercise seriously, questions counted as some part of the course grade, although to what extent was up to the discretion of the course instructor.The assignments were scored after the completion of the term using the rubric included in Appendix A. A score was assigned to responses for each question on a scale of 1 to 4 where 1 is the benchmark response, 2 and 3 are different degrees of milestone responses and a score of 4 was for a capstone response.The scores were compared based on individual questions and on a composite score, which is defined as the sum of the score of Questions 1 to 4. To preserve student anonymity during scoring, all exercises were given a numerical code and photocopied with the student name and class information blackened out.UNC Asheville Institutional Research compiled student information (year in school, declared cluster, cluster courses completed); this information was associated to each code after all responses were scored.
Student responses were scored by four cluster faculty and results were tabulated by numerical code.Two faculty reviewers (LM and DC) scored the answers to Questions 1 and 4, and two faculty reviewers (AL and KP) scored the answers to Questions 2 and 3. Prior to scoring all student responses, inter-rater reliability amongst the four faculty reviewers was determined on six randomly selected student responses.The Kappa level for the four faculty reviewers was 0.79, which is considered to represent substantial agreement.

Statistical Analysis
Only data from students who responded to all four questions were analyzed (n=338).Student responses were analyzed both as composite scores (the sum of the four questions) and individually for Questions 3 and 4. Questions 3 and 4 specifically questioned student learning in Wingert, J.R., Wasileski, S.A., Peterson, K., Mathews, L.G., Lanou, A.J., & Clarke, D.
Journal of the Scholarship of Teaching and Learning, Vol.14, No. 1, February 2014.josotl.indiana.edu49 food literacy, understanding food systems and interest in civic engagement related to food systems, which are the cluster learning goals with the strongest impact based on this team's previous perception study (Wingert et al., 2011).Questions 1 and 2 were not individually evaluated because they are focused on evaluating information rather than the learning goals of the cluster.One-tailed Mann-Whitney tests examined differences between groups of students who had 1) taken versus not taken Food for Thought cluster courses and 2) intentionally declared Food for Thought cluster courses versus those who had not.Spearman's correlation evaluated the relationship between composite response scores and the number of Food for Thought cluster courses taken.The Kruskal-Wallis test evaluated the relationship between composite response scores and students' year in school.The significance level was p < 0.05 and all analyses were conducted using GraphPad Prism 6.0 software (GraphPad Software, Inc.; La Jolla, CA).

Results
Demographic information on the set of student responders (as both number of responses, n, and percent of survey response) is listed in Table 3 and includes the level of college, the distribution of students in control and cluster classes, the number of cluster courses each student had completed or were currently enrolled at the time of conducting the assessment, and the declaration status of each student.The distribution of responses is relatively even across college level (18.9% freshman, 26.9% sophomore, 22.8% junior, 28.4% senior and 3.0% other, which includes non-degree seeking students) and was relatively even between the number of responses from cluster and control (i.e., non-cluster) students (47.6% and 52.4%, respectively) (Table 3).

Comparing students in cluster classes vs. control classes
Response scores were evaluated in several ways.The composite score is the sum of the scores for Questions 1, 2, 3 and 4 for each student and reflects the total performance of the exercise.In addition, individual responses specifically on Questions 3 and 4 were evaluated because these questions specifically targeted the following cluster learning outcomes: food literacy and understanding food systems (Question 3) and civic engagement and informed consumer (Question 4), which all involve integrated learning.Figure 1 displays the difference in (A) composite score and (B) scores for Question 3 and Question 4 between students enrolled in a cluster course (filled bars) vs. enrolled in a control course (open bars).The mean scores, standard deviations (SD) and mean differences are also listed in Table 4: composite score mean difference = 1.02, p < 0.0001; Question 3 mean difference = 0.25, p = 0.002; and Question 4 mean difference = 0.30, p < 0.0001.The mean differences are all statistically significant (p < 0.05).
The composite score is significantly correlated with the number of cluster courses taken, for the full student population assessed and with the control group taking 0 courses (Figure 2A, Table 6; Spearman r = 0.23, p < 0.0001).Therefore, students who take more cluster courses have higher response scores.Responses were evaluated separately for students with intentional interest in the Food for Thought cluster by comparing the n=33 students who had declared the Food for Thought cluster with the control group.Statistically significant differences were measured.Figure 3 displays the difference in composite score (A) and scores for Question 3 and Question 4 (B) between Food for Thought declared students (filled bars) vs. enrolled in a control course (open bars); the mean scores, standard deviations (SD) and mean differences are listed in Table 5.The composite score difference for Food for Thought declared students vs. controls (mean difference = 1.32, p = 0.0008) is greater than the comparison of students enrolled in Food for Thought classes but not declared vs. controls (Figure 1, Table 4).But, similar differences were measured for Question 3 (mean difference = 0.26, p = 0.04) and Question 4 (mean difference = 0.30, p = 0.02).
The composite score was significantly correlated with the number of cluster courses taken by students who had intentionally declared the Food for Thought cluster (Figure 2B, Table 6; Spearman r = 0.32, p = 0.04).The correlation for cluster students is greater than for the group of all students (Figure 2A).There was also a significant positive correlation between response score and number of cluster courses for Question 3 (r = 0.42, p = 0.009) for declared cluster students.However, there was no such significant correlation for Question 4 (Table 6).

Comparing students based on academic maturity
An alternative hypothesis was tested to determine if the positive Spearman correlation for student score vs. number of cluster courses could instead be due to academic maturity.Plotted in Figure 4 is the composite response score vs. year in school for students who have declared the Food for Thought cluster.A Kruskal-Wallis test assessing composite response score differences by year in school found that year in school was not a significant factor.However, year in school was a significant factor (p = 0.012) for the full population of students responding to the questionnaire (not shown here).For the larger sample, score response increased with advancing academic rank.

Discussion
These results demonstrate that the various interactions of the declared cluster students have a statistically significant positive impact on integrative student learning in the areas of food and research literacy, critical thinking about and understanding of food systems, and becoming an informed consumer of food.Students exposed to the Food for Thought cluster, even if only in a single course, performed better on our measures, and their performance increased with increasing exposure via additional courses.Students who declared the food cluster as part of their general education requirement also performed better, suggesting an important role in motivation for performance.Finally, the experience of the Food for Thought cluster enhances learning independently of student maturity.Previous studies reported similar benefits of integrative pedagogies, including the ability to transfer knowledge from one course to another, increased motivation and cognitive ability, and increased responsibility of students towards their learning (Gray, 2000;Tinto, 2000;Inkelas et al., 2006;Stefanou & Salisbury-Glennon, 2001;Avens and Zelley, 1992;MacGregor et al., 2000).Our study contributes to this body of research by examining students' ability to apply knowledge and skills to new material and to do so in an integrative way.Substantive, discipline-based knowledge is mobilized by students to critically evaluate and creatively consider new problems brought to their attention.Our study shows this with regard to measures of exposure, student commitment, and student maturity.Our findings also suggest important strategies for successful integrative curricula.

Impact of exposure to the Food for Thought cluster
Students in the experimental group had statistically significantly higher composite scores on our assessment instrument than did students in the control group; this finding holds true when the scores of the two groups are compared only on assessment Questions 3 and 4, the essay portion of the assessment instrument (Figure 1).Taking a course in the Food for Thought cluster means that students were better able to critically evaluate a news article on a global food related issue than those who had not had a course in the cluster.It also means that the enrolled students were better able to identify interdisciplinary connections regarding a food issue and to better articulate a reasoned approach to how they consume food.
Moreover, the scores on our composite measure increase with increased exposure to the Food for Thought Cluster curriculum, as measured by number of courses taken (Figure 2A).This means that taking more cluster courses increases students' ability to identify interdisciplinary connections regarding a food issue and to better articulate a reasoned approach to how they consume food.

Impact of intentional selection of the Food for Thought cluster
Because some students are enrolled in Food for Thought courses without necessarily having the intent of completing the cluster of three courses, we also compared the scores of those who had declared the cluster and completed at least one course with those who had not declared or enrolled in a Food for Thought course.The same trend holds here.It appears that students' commitment to the cluster as a part of their undergraduate education enhances their learning attainment in the cluster (Figure 2B); and the more courses that declared students take, the higher their composite scores and the higher their scores for assessment Question 3. A similar correlation was not found for Question 4, perhaps because the question is about consumer choices and students committed to the cluster may develop this sensitivity early on in the cluster or have some predisposition to consumer consciousness prior to enrollment in the cluster (Table 6).Our interpretation is that identifying the interconnectedness of disciplines is a skill that students develop over time as their exposure to multiple disciplines within the cluster grows.

Impact of student maturity
In the full sample of participants in our study, students with higher academic rank scored higher on the composite score than students with lower ranks.Among the sample of students who had declared the cluster, however, year in school did not have a significant effect on scores (Figure 4).We conclude that engagement in the cluster curriculum enhances student learning independently of student maturity.

Implications for curriculum planning
Our data support the notion that integrative learning strategies across courses yield positive learning gains.During the course of our first study and this second study, the specific cross-course projects changed-but positive results were found in both instances.The second set of course projects were adjusted to simplify the structure and coordination of tasks necessary to faculty, reducing the amount of faculty coordination time necessary.The lesson we take from this is that even relatively simple efforts at having students connect across courses are worthwhile for student learning.The recognition that even simplified and focused projects can create meaningful learning is especially important now, when budgets are tight.It is indeed possible to realize significant gains in student learning from less time-and money-intensive (but no less intentional) collaborations across classes.This is good for students (less time commitment for them, potentially fewer complaints about group projects), faculty (less time to coordinate), and administrators (smaller budget is needed to achieve similar learning gains) alike.

Conclusion
Our previous study measured students' perception of their learning gains; the study reported here confirms students' perceptions by measuring actual learning gains.Exposure to multiple disciplines covering a shared topic enhances students' ability to integrate diverse forms of knowledge and to see an issue from multiple perspectives-an essential goal of liberal arts education and interdisciplinary learning.Undergraduate curricula that allow for this kind of cross-fertilization provide for more holistic ways of identifying problems and strategizing solutions.They create more flexible thinkers in our students.At a time when the liberal arts is under attack for not being practical or leading to immediate job readiness, our findings suggest that the kind of employees sought by many business leaders (employees who can see things from multiple perspectives and who are problem solvers with multiple tool kits), our data is especially significant.The findings also support the notion that integrated learning requires some time in

Figure 1 .
Figure 1.Differences in (A) composite score responses and (B) Question 3 and Question 4 score responses between students who have not taken a Food for Thought Cluster course (control, open bars, n = 170) and students who have completed at least one course in the cluster (cluster course, filled bars, n = 168).Mean score and standard deviation are shown.< 0.05.

Figure 2 .
Figure 2. Spearman correlation between composite response score and the number of Food for Thought Cluster courses for (A) all study participants, and (B) only students who had intentionally declared the Food for Thought cluster.

Figure 3 .
Figure 3. Differences in (A) composite score responses and (B) Question 3 and Question 4 score responses between students who have not taken a Food for Thought course (control, open bars, n = 170) and students who had intentionally declared the Food for Thought Cluster and completed at least one course in the cluster (cluster 9, filled bars, n = 33).Mean score and SD are shown.*P < 0.05.

Figure 4 .
Figure 4. Response score differences across year in school for students who had intentionally declared the Food for Thought cluster.

Table 2
Food for Thought Cluster Projects and ActivitiesAY 2011AY  -2012

Table 3
Demographic information about students responding to the questionnaire

Table 5
Response score differences between groups, declared cluster students vs. control