Integrating and Assessing Essential Learning Outcomes: Fostering Faculty Development and Student Engagement

: This paper proposes a model for implementing a college-wide initiative designed to promote student competence with essential learning outcomes. This mixed-methods study combined descriptive and qualitative approaches to explore experiences of students, faculty, and professional as they engaged in a project that focused on teaching and learning with essential learning outcomes (ELOs). The researchers relied on the use of descriptive statistics and interpretive phenomenological analysis to capture faculty and professional staff participants’ engagement in a collaborative study group designed to support pedagogy integrating ELOs into courses, assignments, and co-curricular activities and to gauge students’ experienced competence. Six themes emerged from the focus on ELO integration, including connection, awareness, utility, reflection, intention and facilitation. Additionally, using a pre and post student survey of self-perceived ELO competence, researchers found a difference in student ratings of 0.25, (p ≤ 0.05), suggesting an increase in students’ perception of their ELO competence over the course of the semester.


Introduction
Many college and universities have responded to accrediting organizations' requirements for student outcomes assessment by establishing institutional goals.This requirement, coupled with employers' demands for colleges and universities to prepare graduates with practical skills, professional knowledge, and intellectual abilities, has created challenges for higher education in the United States.To assist liberal arts institutions establish assessment practices and collection of evidence for direct assessment of student learning that satisfy both accreditors and employers, the American Association of Colleges and Universities (AAC&U) launched its Liberal Education and America' s Promise (LEAP) initiative in 2005 (https://www.aacu.org/leap).Since then, over 100 liberal arts colleges and universities have adopted LEAP essential learning outcomes (https://www.aacu.org/leap/essential-learning-outcomes)and its associated Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics (https://www.aacu.org/value).Although informed by AAC&U's leadership in outcomes assessment, the public liberal arts college in this study identified ten essential learning outcomes (ELOs) specific to curricular and co-curricular learning experiences offered on its campus.Some of the resulting ELOs overlap with LEAP's, however, not all.As well, this college embraced institution-wide participation in the design and implementation of its ELO initiative rather than creating an administratively driven compliance model to address accreditors' requirements and employers' demands.This approach has led to gradual, but increasing, campus-wide commitment to the ELO initiative; however, it has also necessitated ongoing professional development.The study described in this article presents the Cydis, S., Galantino, M. L., Hood, C., Padded, M., & Richard, M. findings from one professional development opportunity and offers a model for learner-centered implementation of ELOs in courses, assignments, and co-curricular activities.

Background Essential Learning Outcomes
Shifting the current model of education to one that includes a learner-centered, competency-based approach to education is an important priority for reform in the 21 st century.(Altbach, Reisberg, & Rumbley, 2010;Barber, Donnelly, & Rizvi, 2013).While there continues to be a growing emphasis on quality pedagogy, colleges and universities must also create cultures that value student learning outcomes in their efforts to ensure institutional effectiveness (Kuh, Jankowski, Ikenberry, & Kinzie, 2014).LEAP's essential learning outcomes (Figure 1) represent one national project supporting this model of outcomes-focused and competence-based student learning.
The LEAP National Leadership Council (2007) refers to a "Framework for Excellence," which encompasses the process of integrating ELOs across disciplines through an intentional focus on the part of teachers and students.As part of the LEAP initiative, AAC&U also promotes the importance of "High Impact Practices" to support the students' acquisition of essential learning outcomes.The council emphasizes the importance of offering students opportunities to apply interdisciplinary knowledge and to acquire a perspective for learning that involves creating knowledge through real-world problem-solving activities.Teaching and Learning, Vol. 15, No. 3, June, 2015. josotl.indiana.eduimplemented effectively with an (10) emphasis on evaluation.This model, although developed for an online environment, is a meaningful framework for face-to-face classroom instruction as well.

Journal of the Scholarship of
A competency-based educational model proposed by Kovacs, Hutchinson, Collins, and Linde (2013) focuses on outcomes and levels of competence developed through students engaging in and performing real-world tasks.The model includes a three-step process in which educational goals are stated as competencies students acquire through course learning and assessment tasks.
Step two requires procedures for assessing achievement of competencies, and step three focuses on the learning experiences that build competencies in students.

The Science of Teaching and Learning: A New Paradigm for Teaching and Learning
In a survey conducted by the National Institute for Learning Outcomes Assessment, researchers found that assessment efforts centered on yielding meaningful measures of student learning and documenting student accomplishment are most valued by provosts, rather than those measures that interest the government or accreditors (Kuh, Jankowski, Ikenberry, & Kinzie, 2014).Walker (2008) eloquently described a phenomenon that requires careful consideration of the unintended, critical impact of a model for learning that has prevailed, possibly as a result of a bureaucratic structure in which the institution of education exists, "over the centuries, we have refined our definitions of learning to mean a certain kind of school learning, and educational systems have been geared to nourish a narrow range of human talent which, when defined as "the ability to manipulate the abstractions of academe (p.12), is estimated to be one-tenth of human ability," (Cross, 1976, p.12;Taylor, 1968, as cited in Walker, 2008).
However, efforts toward a competency-based approach were evidence of a change in the way we approach learning and assessment in the classroom, one that can be thought of as a "New Paradigm for Teaching and Learning" (Barr & Tagg, 1995).Barr and Tagg describe a model of education that is focused on the learner rather than the instructor.The model asserts to improve the quality of learning through powerful learning environments in which learners are encouraged to discover information through learning experiences designed to support student competencies.Learning is constructed by the student and is shaped by individuals' experiences.The criteria for success integrate quantity as well as quality student learning outcomes.Learning structures are integrated and holistic and are centered on the student through active engagement with the content.Through this model, ultimately the degree equals demonstrated knowledge and skills.Using this model, students have the opportunity to document quality learning through the significant learning experiences (Fink, 2013) that demonstrate student competence with specified essential learning outcomes.Artifacts then serve as quality learning and evidence of that competence.A focus on the construction of significant learning experiences (Fink, 2013) offers a model for instructors to support student competence by designing instruction that resides within the learning paradigm.

Significant Learning Experiences
Fink (2013) identifies taxonomy for significant learning and a model for integrated course design as part of a learner-centered approach to instruction.The taxonomy includes six components upon which the design for instruction is based.Foundational knowledge is part of the content which students need to learn, an essential part of any course and important to the program outcomes students need with any degree.Application serves as a second component in which the learner applies content through integration, which is identified as a third component of the taxonomy.This component offers students the opportunity to integrate learning through engaging in authentic meaningful tasks that are personally relevant to students as part of a focus on integration of knowledge and abilities for depth of understanding.Through that engagement in authentic meaningful tasks, students acquire a commitment to the learning, which Fink describes as caring, a fourth component of the taxonomy.Fink refers to the human dimension as a fifth component, and acknowledges the role that the learner's perspective plays in terms of the learning experience as well as the perspective of others.Lastly, learning how to learn describes a component of the taxonomy that focuses on the learner's awareness of the skills and abilities they have acquired through the learning experience.This awareness on the part of students is an important consideration as it relates to supporting student competence with essential learning in higher education.

The Role of the Student: Student Awareness and Perceptions of Learning
Student self-awareness of learning has the potential to support student competence with essential learning outcomes.The learner's self-awareness of their learning can be described as metacognition.Metacognition is defined by Flavell (1976) as, "one's knowledge concerning one's own cognitive processes or anything related to them."Metacognitive awareness can become a powerful learning strategy for students and has the potential to further support life-long learning (Wehlburg, 2006), what one might also consider essential learning outcomes.With metacognitive awareness of ELO competence, students can potentially acquire a greater understanding of the skills they possess and engage in cognitive thought processes to apply those skills in the life and workplace experiences they encounter.
Relying on student perception of metacognitive awareness can play an important role in supporting students with developing ELO competence.In a study of student perceptions of learning, Walker (2008) found that student perceptions of learning revealed learning well beyond traditional course content.Students identified areas of competence that parallel those identified by LEAP as essential learning outcomes.In this sense, student reflection demonstrates validity as a direct measure of student learning related to essential learning outcomes.Recognizing the important role of student perception of learning has the potential to enhance and support other direct measures of student learning.Other direct measures of learning, such as performance based tasks, offer the opportunity for students to reflect, analyze and interpret concepts acquired and facilitate awareness of essential learning outcomes that go beyond the content of the course (Walker).
Spronken-Smith et al. ( 2012) conducted a study in which they surveyed students as to which course learning and assessment activities students perceived fostered learning on various levels and recommended more structured opportunities for reflection to support student learning.This serves as evidence of the value of teachers who intentionally focused on informing students of the range of course outcomes and supporting student awareness of their own learning to foster deeper learning processes.
Encouraging students to reflect and develop self-awareness of the Essential Learning Outcomes targeted in their coursework has the potential to support competency and the readiness needed for success in the 21 st century and a global economy.Engaging students in the practice of attending to the learning acquisition and promoting opportunities to develop metacognition may further support students' ELO awareness as they develop strategies for life-long learning.Given Journal of the Scholarship of Teaching and Learning, Vol. 15, No. 3, June, 2015. josotl.indiana.eduthe value of ELO competence and the role that student awareness of ELO competence has in preparing students for post-secondary arena, this study served as a valuable opportunity to assess students' level of ELO competence acquired over the course of the semester while enrolled in a class in which the instructor focused on ELO integration.Furthermore, students' awareness of ELO competence was assessed through a written statement in which students communicated perceived competence in the ELOs targeted for instruction.

Study Design
The design of this study is largely descriptive and qualitative in nature and included the use of Interpretive Phenomenological Analysis (IPA), a process of analysis used to identify the phenomena that emerge, recognizing the role of the researcher's interpretation of the data (Reiners, 2012).Methodology surrounding IPA includes interpreting data and uncovering thematic aspects of the experiences investigated in the study.Researchers are not independent of the meanings extracted from the data, but instead are part of the phenomena.In this study, the researchers sought to explore the phenomena surrounding students', faculty members', and professional staff members' experiences integrating ELOs into courses, assignments, and co-curricular activities.While faculty and professional staff conducted their courses and co-curricular activities, researchers sought to identify themes that emerged from their experiences.The descriptive portion of the study involved pre and post surveys of students' perceived competence with the ten ELOs identified by the college (http://www.stockton.edu/elos,Appendix A).

Setting/Participants
A public liberal arts college in a suburban area was the venue for data collection.The college enrolls approximately 8,570 and offers baccalaureate and graduate level programs in the liberal arts and professional studies.The majority of undergraduates are full-time, matriculated students.99% are from the state where the college is located.75% are Caucasian.The male to female ratio is 40% to 60%.35% of the students receive Pell Grants, and 76% borrow at some point during their college education.The college has 304 full-time faculty members.Twice as many are associate as full or assistant professors, and two-thirds of the faculty members are tenured.The sampling method used to solicit participants was criterion-based, non-probability sampling.This type of sampling "requires that one establish the criteria, bases or standards necessary for units to be included in the investigation, one then finds a sample that matches these criteria" (Merriam, 1988, p. 48).After approval from the Institutional Review Board, the researchers sent out a call for faculty and professional staff participants through the College's email system.The call described the study group project and encouraged interested faculty and professional staff members to write a letter of interest to participate in a study group for the purpose of learning to integrate ELOs into courses, assignments, and co-curricular activities.Nineteen faculty and professional staff members were selected.In addition to attending study group meetings, they volunteered to share student data with the study group facilitators (the researchers) and to attend a focus group at the end of the semester.The study group began during the mid-semester break in 2013 and extended through the spring semester 2014.scholarship on essential learning outcomes, course and co-curricular design, and integrative learning.The goal of these meetings was to encourage participants to practice intentional revision of their pedagogical practices, to incorporate ELOs into their course and co-curricular goals, and to associate relevant ELOs with particular learning experiences.In addition, participants were asked to design one learning task intended to elicit one or more ELOs.The researchers collected data about the effectiveness of these meetings through participant interviews and focus groups and students' reflections.
Faculty and professional staff participants selected the courses and co-curricular experiences targeted for ELO integration.Learning experiences ranged from those aimed at first through fourth year courses and co-curricular activities, and general studies courses to courses in.business, social sciences, and nursing.Student participants included those enrolled in the courses taught by faculty participants or those who attended co-curricular activities facilitated by the professional staff participants.
The researchers collected additional data of students' perceived competence with ELOs.Students were given pre and post surveys for self-reporting.The researchers also collected qualitative data from students' end-of-semester reflections.

Data Collection Procedures
Consent.Participants and students signed IRB-approved informed consent forms, agreeing to take part in interviews and focus groups (participants) and to complete pre and post surveys (students).Five faculty participants in the ELO study group and their students consented.Professional staff participants in the ELO study group did not consent, primarily because the ELO study group occurred after their co-curricular activities had already taken place.As a result, they joined the ELO study group to gain skills for ELO implementation in future activities.In total, the researchers collected data from 111 students enrolled in seven courses.
ELO Student Perception Survey.At the beginning of the semester, after explaining the study and obtaining written consent, researchers read aloud the "Administration Instructions -ELO Survey" (Appendix B) to students who chose to participate.Researchers distributed the Essential Learning Outcomes Survey (Appendix B) to students.The survey asked students to rate their perceived level of competence in each of Stockton's ten ELOs using a 3-point Likert Scale.Levels of competence included (1) aware, (2) competent and (3) skilled.These data were collected as a pretest to determine students' baseline perceived level of competency.As the semester progressed, students engaged in learning and assessment tasks designed by participants that were designed to elicit specific ELOs.
At the end of the semester students completed the same survey.Post-test data were compared to pretest results and analyzed using the paired t-test to determine changes over time and as a result of intentional ELO integration into courses and assignments.

Student Written Statements.
Researchers provided faculty participants with the writing prompt shown in Appendix C. In addition, students were prompted to reflect on the important ELO outcomes of the course and to respond to the following questions: "What have you learned with regard to these learning outcomes?"Faculty participants provided these questions to the students in their courses to capture written statements describing their perceived level of ELO competence at the end of the semester.A content analysis of this data involved coding and thematic analysis to identify themes relating to students' metacognitive knowledge of their competence in ELOs.
Faculty Interviews.Researchers conducted individual interviews with faculty participants to investigate phenomena surrounding student awareness of ELO competence.Each faculty member was asked to consider the following question "How aware do you think students are about the ELOs they acquire from experiences both in and out of the classroom?"These interviews were transcribed, and the researchers performed a content analysis of the interview data, which involved coding and thematic analysis to determine emergent themes.

Data Analysis
Thematic Analysis Procedure.Faculty interview data were transcribed and coded to identify emerging themes using the process of Interpretive Phenomenological Analysis.This process included a six-phase analysis of qualitative data as identified by Braun and Clarke (2006).During the first phase, researchers transcribing the data, and read through the data a number of times.In the second phase, researchers used HyperRESEARCH (2014) to generate initial themes.Then, they tentatively aligned passages from the interviews with relevant themes.In the third phase, researchers reviewed the alignments and made any necessary adjustments matching passages to themes.The researchers then reviewed the data to ensure consensus (fourth phase) and organized the themes into a meaningful visual representation (fifth phase).Finally, the researchers refined the specific components of each theme, clearly defined each theme, and identified the themes on the visual representation in such a way that the visual representation communicated the findings of the study group (sixth phase).
Student Statement Content Analysis Procedure.The content analysis process used with the reflective statements written by students included, first, transcribing the reflective statements (first phase), and reading the statements a number of times (second phase).In the third phase, using N-Vivo (2008), researchers highlighted instances in which students identified learning related to any one of the ELOs.The researchers, then, reviewed the data to ensure consensus (fourth phase).Finally, compelling examples of each theme were extracted from the student data to support the findings of the study group (fifth phase).It was not necessary to construct a separate visual representation to communicate these findings, as these results did not require any modification of the previously constructed visual representation (sixth phase).

Results
ELO Student Perception Survey.Findings indicated that overall scores on the post-test were higher with a mean difference of 0.25 (p ≤ .05).These results show increase in student perceived ELO competence over the course of the semester.The differences in the means for each ELO in each course showed increases ranging from 0.16 to 0.52 (Figure 2).Furthermore, the respective means of the ELOs targeted by faculty participants for integration into their respective courses showed an increase at a level of significance (Table 1).Teaching and Learning, Vol. 15, No. 3, June, 2015.josotl.indiana.eduThemes Identified in Student Statement Data.The thematic analysis of student statements focused on references to ELOs as interpreted by researchers.Table 2 reflects the results which included between 10 and 36 references for each ELO targeted for instruction.

Journal of the Scholarship of
Students cited many examples of growth in the areas of these ELOs.For example, students' responses included references to the ability to think beyond the scope of the class as evidenced by the following statements.

"I have learned how to think and create beyond the scope of the class to achieve an open-ended goal."
"It gave some perspective to the theories and knowledge we have learned by applying it to real world scenarios."Students also identified the need to integrate the use of critical thinking skills with other information and settings as evidenced in the following examples.
"I have learned not to take everything I read at face value, but rather to compare what I read to my own personal views and to consider why a particular culture feels a certain way about a topic."As examples of the growth in the area of Ethical Reasoning, students connected this learning outcome to the pragmatic application of these skills as in the following examples.
"Having these skills will give me the knowledge to find and use appropriate resources and the confidence to question practices not based on evidence."

"It is important to express your feelings and view others to gain experience and knowledge not just your own interpretations."
These examples of self-described ELO acquisition on the part of students serve as another measure of student learning and further evidence of increased achievement in ELO competency.
Themes Identified in Faculty Data.As a result of the thematic analysis of the faculty interview data, six themes were identified from shared perspectives of faculty participants.The following are features of ELO integration that support student competence with ELOs: Facilitation.This theme includes a reference to faculty members' comfort level or knowledge about ELO instruction and the role that teaching plays in supporting student competency with ELOs.This theme might also identify the need for faculty to guide or facilitate ELO acquisition in students through planned or incidental advising.
Intention.Faculty participants identified a need or value for explicit emphasis of ELOs on the part of instructors to support student understanding.References to this theme include statements regarding repeated presentations of ELOs to students, revisiting the ELOs with students and/or providing continuity in discussing the ELOs with students.
Connection.This theme includes instances in which students connect one of the ELO to either another ELO or possibly connect one ELO intellectually or experientially over time.For Journal of the Scholarship of Teaching and Learning, Vol. 15, No. 3, June, 2015.josotl.indiana.eduexample, appreciating ELO knowledge from the beginning of the semester to the end of the semester.It might also include students recognizing continuity or a recursiveness in one or more ELOs.Finally, this theme included statements referring to linking concepts in a manner that demonstrates one or more ELOs.These concepts might also include course content or concepts learned in pragmatic experiences.
Utility.In regard to this theme, faculty referred to the intended use of the ELO, a value for the authentic purpose of the ELO in the future.It might also refer to preparation of and/or application of the ELO beyond the scope of the classroom.This theme included references to the ELOs and life-long learning.
Reflection.According to Boud, Keogh, and Walker (1985, p. 19), "reflection in the context of learning is a generic term for those intellectual and affective activities in which individuals engage to explore their experiences in order to lead to new understandings and appreciations".For this reason all references that coincide with this definition were identified within the theme of reflection.Additionally, references to this process of reflection and documentation of the behavior were also considered references to this theme.
Awareness.This theme emerged from statements that illustrated students' ability to identify one or more ELOs through a learning experience or a life experience, which heightened or contributed to recognition of ELO competence.This experience was interpreted as one that illuminated students' abilities as it related to ELO competence.
Themes Identified in Faculty Data.As a method for ensuring the validity of the themes that emerged from this data, the researchers asked faculty participants to review the themes and comment on the credibility of the researcher's findings.Faculty participants were asked to review the six themes and the definitions identified by the researchers as outlined in the discussion above.As a result of this check of the validity of the data, faculty participants reported that the themes identified were either credible or very credible interpretations of the data.

Findings
This mixed methods research revealed that several components of ELO integration converge to support student competence.Overall, students showed significant growth in all ten ELOs.This finding was further confirmed by student qualitative statements to further probe the level competence as it related to the ELOs.The phenomena that emerged from the faculty interviews included six themes as instrumental to supporting student competence with ELOs.
Faculty identified the importance of (1) facilitating ELO integration by developing a comfort level or knowledge base with ELO instruction and recognizing the role of faculty in guiding and facilitating learning through planned or incidental experiences to effectively support ELO acquisition in students.Faculty have the opportunity to facilitate this phenomenon through explicit emphasis and (2) intentionality of ELOs on the part of instructors through repeated presentations with these in mind to students, revisiting the ELOs with students, using ELO vocabulary and/or providing continuity in discussing the ELOs with students has the potential to support ELO acquisition in students and support metacognitive awareness..This further supports the phenomenon of (3) connection as another theme that emerged from this study.As students engage in learning experiences to support ELOs, the findings suggest that student learning might benefit from a focus on making explicit connections among ELOs and opportunities to see relationships between them and the practical settings for which they are intended.engagement with the content, shaped by students' individual experiences.This concept of connection can also be illustrated in Shavelson's (2007) model as the need for learning that is highly situated within a meaningful context, learning that includes extensive engagement, deliberate practice and formative feedback to support students' ability to transfer the learning to new situations.This concept is further supported by Voorhees' model which emphasizes the importance of competencies being linked and embedded with courses across disciplines.This forms the foundation for learning that students acquire upon which further learning is built.MacLean and Scott (2011) reference this concept in their model which emphasizes student understanding of the "processes of change" and parallel the theme of connection as students connect concepts related to competencies over time.
The concept of awareness as an emerging theme illustrates the important role that metacognition has in ELO competence.Metacognitive awareness can become a powerful learning strategy for students.It has the potential to further support life-long learning (Wehlburg, 2006), and is therefore an important component of this model of ELO integration.Fink (2013) emphasizes the importance of learning how to learn.This is a good illustration of the role that metacognitive awareness plays in this model.Voorhees (2001) identifies the importance of making students aware and reflective about the criteria for evaluation to support competence in students.He asserts the importance of faculty unambiguously identifying student learning outcomes and articulating specific competences as faculty guide to facilitate ELO acquisition in students.Through planned or incidental advising, students have the opportunity to build metacognitive awareness and further develop ELO competence.MacLean and Scott (2011) also recognize the importance of facilitating the learning process through effective organization of the course that includes clearly stated learning outcomes, a concept also included as part of the model proposed by Journal of the Scholarship of Teaching andLearning, Vol. 15, No. 3, June, 2015. josotl.indiana.edu Kovacs, Hutchingson, Collins, andLinde (2013) who identify specifying educational goals explicitly as part of their three step process to supporting competence.This also supports an intentional focus, another emerging theme in this study.Faculty harness an intentional stance as they engage in the explicit identification of learning outcomes and facilitate learning on the part of students.
Through an intentional stance, faculty and students have the opportunity to further support learning through practices such as explicitly identifying in the syllabus, opportunities for supporting students' competence with ELOs (Voorhees, 2001).MacLean and Scott (2011) also note the need for faculty to clearly state course aims and learning outcomes and provide formative feedback in an effort to support student learning.For example, as a student engages in a learning experience that may have been designed with the intention to support a particular ELO, the student might gain a greater awareness about the specific use of the ELO, or a greater awareness of the value for the authentic purpose of the ELO in the future, or purpose beyond the scope of the classroom.
This has the potential to further support student understanding by supporting their awareness of the utility of a particular ELO.The theme of utility appears to parallel, what Fink (2013) identifies as the integration of the concepts among the course, other courses and students' personal, social and/or work life.To support competence with an ELO, students engage in learning experience intended to have meaning surrounding the authentic context in which it is intended to be utilized (Voorhees, 2001).MacLean and Scott (2011) recognize the need for course content to be meaningful to the context, interest and personal goals of the learners and emphasized the need for learning experiences to be relevant.This concept of utility is further developed as students engage in and perform real-word tasks, an important part of a competency-based model for supporting student learning outcomes (Kovacs, Hutchinson, Collins, & Linde, 2013).Engaging students in meaningful, authentic, performance-based tasks has shown to support student selfperceived competence (Cydis, 2014).
As students engage in the significant learning experiences, facilitated by faculty and develop the awareness of the utility of any given ELO, students and faculty created opportunities for reflection to further support ELO competence.These opportunities for reflection can be intentional as well as unintentional.Students identified more effective learning experiences given more opportunities to intentionally engage in reflection (Spronken-Smith et al., 2012).Through the emphasis and integration of meaningful learning and providing students with the criteria for evaluation, students have opportunities to build awareness and reflectiveness about their competence (Voorhees, 2001).The process of reflection offers students opportunities to rely on aptitude, natural endowments and abilities to facilitate the acquisition of student learning outcomes and effectively transfer the learning to new situations (Shavelson, 2007).This is also further supported by the human dimension in Fink's (2013) taxonomy in which students have the opportunity to learn about oneself.As students engage in reflection they have the opportunity to learn more about the competence they have acquired.

Conclusions
The results of this study confirm important components of a model for ELO integration into courses, assignments, and co-curricular activities.While the components of this model for ELO integration are interconnected, each also has an independent role in teaching and learning Journal of the Scholarship of Teaching and Learning, Vol. 15, No. 3, June, 2015. josotl.indiana.eduwith ELOs.The components of the model that emerged through this study are: connection, awareness, utility, reflection, intention and facilitation.Students, faculty, and professional staff noted that learning increased when these components were present in course/co-curricular activity design and in assignment and activity descriptions.For instance, students enhance their familiarity with ELOs through exposure to the vocabulary and continued conversation about ELOs throughout the semester.However, students may not be cognizant about integration as they engage in course work designed to support ELO competence unless their teacher intentionally discusses ELOs.Therefore, teachers can enhance student awareness by engaging in explicit conversations about ELOs, revisiting discussion about how learning experiences support ELO competence, and providing students with visuals that clearly articulate and describe ELO competence.It is important to note that a teacher's intentional focus on ELOs does not by itself increase students' ability to develop ELO competence.In the context of an intentional pedagogy, students must also engage in reflection and metacognitive awareness of ELO competence in order to most effectively develop ELOs.However, faculty and professional staff can engage in practices that are likely to lead to development of students' ELOs such as explicit articulation of ELOs in the course/cocurricular activity syllabus, identification of relevant ELOs in learning tasks, and evaluation of the demonstration of ELO learning in assessments.Faculty and professional staff can also provide various opportunities for students' reflection of ELO competence.Finally, students, faculty, and professional staff can work collaboratively to identify and illuminate examples of student ELO competence exhibited in samples of reflective writing, meaningful research and analysis, and projects drawn from real-world experiences.
While each of these themes operate in a very interdependent manner, it is important to note that the relationship among them appears to have the potential to create a synergetic dynamic and ultimately support and facilitate the impact of the other components.As the findings suggest, it is difficult to separate emergent themes as phenomena that supports effective ELO integration and student development of ELOs.The researchers propose that there is an interdependent relationship among these concepts and the synergetic relationship among these features contribute to an effective model of ELO implementation in an educational setting such as the one described in this study.Furthermore, this might also include a recursive nature of ELO competence on the part of the students.
This phenomena is consistent with Fink's Taxonomy of Significant Learning (2013), in which the components of the taxonomy are interdependent and do not exist as a hierarchy, but rather as a relationship that is recursive in nature.Figure 3 represents the interdependent relationship among the phenomena that emerged in this study and were shown to contribute to ELO competence in students.Finally, the theme of reflection emerged and appears to integrate very meaningfully within this synergetic model for supporting ELO integration.

Limitations
Researcher participants in this study were solicited based on their interest in methods for effective integration of ELOs in courses, assignments, and co-curricular activities and the value they placed on the conceptual models outlined in the literature review.For this reason, researcher and participant biases may have played a role in the results of this study.In future studies, researchers could consider adopting control group procedures for ensuring the credibility of the themes that emerged participant data.Additionally, instructor bias should be considered in relation Journal of the Scholarship of Teaching and Learning, Vol. 15, No. 3, June, 2015. josotl.indiana.eduto the effect it may have on the metacognitive awareness students develop.As instructors discuss the concept of ELO competence and the importance of promoting ELO competence it may have an effect on students' perceived ELO competence and this factor should therefore be considered.Creating a control group might also help to mitigate any effect resulting from students' desires to craft reflective statements, or respond to the ELO survey with a perception of increased ELO competence to please their teacher.
The Hawthorne (Shuttleworth, 2009) effect may have played a role in the results of the post-test.At the end of the semester, when students completed the post-test, they were aware that their teacher was participating in a study group to investigate ELO integration into their course and assignments.Consequently, students' knew that their teacher sought to improve their ELO competence.To avoid this effect, future studies that include a control group with students from courses not intentionally integrating ELOs into courses and assignments would offer researchers an opportunity to compare reflective data and to draw more confident conclusions about the effectiveness of the ELO integration model.

Future Research
In future research, the researchers intend to collect direct measures of student learning including artifacts demonstrating ELO competence.These artifacts could be analyzed to ascertain evidence of ELO competence and the extent to which student performance correlates with their self-perception of ELO competence.The researchers will need to create an assessment plan to evaluate the student artifacts.
Further research is needed to learn the specifics of the intentional practices used by faculty to foster ELO competence in students; this aspect of participants' classroom work was not investigated as thoroughly as necessary to fully explore the efficacy of the study group and the ELO integration model.In addition, the researchers need to develop an assessment plan to evaluate the model proposed in this study, including an analysis of the extent to which the themes are interdependent.The researchers also intend to conduct a content analysis of syllabi designed by participants for their communication about ELOs and teachers' intentional ELO focus.
Finally, the researchers aim to analyze the usefulness of future innovations to the construction of the model of ELO integration presented in this study and to the assessment of the model's effectiveness as a means to increase students' learning and ELO competence.The researchers might consider experimenting with the Degree Qualifications Profile (Lumina Foundation, 2013) as a way to connect outcomes students acquire at the course level to the outcomes identified on such a national initiative.The model presented here is flexible enough to integrate those innovations, especially those that align with the college's ten ELOs and that promote the most successful student learning experiences.

Table 1
Difference in Pre and Post Means for each ELO

Table 2
Frequency of References to ELO Competence in Student StatementsStudents described Global Awareness skills as those that include a value for understanding various perspectives and the ability to better understand and interact with people as a result of these new understandings: "Learning these skills helps in any setting, academic or professional, and helped me to identify more than my own point of view and how to be able to adapt to my surroundings."Theintegratedrolethatteamworkplays in meeting with success in group settings is evidenced by the following statements."I'velearnedhowtoworkwith a group and bounce ideas off of those around me to better my skills and what I'm working on.This is important because, in the field of Programming, people often work in groups to get a code done and it's crucial to make sure everyone is on the same page and has the same idea about things.If you can't communicate clearly, you are not going toJournal of the Scholarship ofTeaching and Learning, Vol. 15, No. 3, June, 2015.josotl.indiana.edu"….domy job independently and learn, evaluate, and react to new challenges in this field.""Inan ever-growing culturally diverse society, it is important that I make myself aware of the nuances of cultural groups to eliminate any biases or prejudices that I may have toward them."