A Curricular Intervention for Teaching and Learning : Measurement of Gains of First-Year College Student Learning

There is continued debate about the changing and varied nature of first year college courses in American higher education institutions; therefore, additional research is needed to inform faculty and administration of the specific areas on which to focus for teaching and learning in these courses. This study is one possible approach to easily determine student “gains” as measured by the College Success Factors Index, or CSFI. This index consists of ten college student success factors that indicate student achievement. The purpose of this research was to examine the CSFI pretest and post-test data for possible student gains. These selfevaluations were administered in both the summer 2013 and summer 2014 semester courses. The CSFI pre and post-test scores comparing each summer were analyzed and the overall gains were observed. The summer 2013 courses were taught without a focus on the targeted success factors. A curricular intervention was then introduced in the summer 2014 courses that integrated the ten success indices into the lessons, activities, and assignments. The overall differences or “gains” in student learning are described, along with implications for first year teaching and program design in colleges and universities.


Introduction
The design and subsequent continuous improvement of a first year experience course that maximizes student learning and success is a very complex endeavor.Depending on the changing priorities or administration, level of student aptitude, faculty skill/experience level and a host of other factors, first year experience courses have a tendency to be placed in a certain category.The 2009 National Survey of First Year Seminars (Padgett and Keup, 2011) has identified five common types: (a) extended orientation, (b) academic with uniform content, (c) academic on various topics, (d) pre-professional or discipline linked, and (e) basic study skills.
The first year experience course that was examined in this study is a life skills/study skills college success course offered at the University of Central Florida with the course number and title of SLS1501: Strategies for College Student Success.This class followed the traditional approach of presenting the learning topics and issues based on research about first-year college student achievement.The teaching of these important topics is not based on opinion or simple advice, but rather rigorous research related to college student success; therefore, the course is an academic workshop with uniform content.
In an effort to improve, innovate, and assess the effectiveness of the summer of 2013 workshop, the steering committee for the course required faculty in all sections to administer the College Success Factors Index, or CSFI.The CSFI will be described in greater detail throughout

Success Factors
"Responsibility/Control" over assignments and other accountabilities are associated with attribution theory.There are two psychological constructs related to this first student success factor: locus of control and motivation.Locus of control consists of two facets: internal locus, where students attribute their successes and failures in college to their own efforts and motivations, and external locus, believing their outcomes are due to chance.According to Bean and Eaton (2001), students with an internal locus of control connect their academic and social achievements to studying and attending classes which bolsters their motivation, while others with an external locus believe that their achievements are related to luck or professors 'liking' them.Bean and Eaton (2001) stated that student efforts and psychological responses to their interactions with the college lead to academic and social integration.The external locus of control mindset can severely

Cox and Lemon
Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 3, June 2016.josotl.indiana.edulimit college students' motivation to take responsibility for their academic and social achievements, thus limiting their chances of integration, which plays a fundamental role in retention and persistence to graduation.
"Competition", between other college students or internally, make up the second student success factor.This is related to academic and social integration."The more students are involved in the social and academic life of the institution, the more likely they are to learn and persist" (Tinto, 1998, p. 2).Students need to forge relationships with fellow classmates and faculty and engage in academic discourse.According to Zhao and Kuh (2004), an excellent way to build academic and social integration is to have college students participate in learning communities; the results from their research indicate that learning communities are linked to greater academic effort, better academic integration, more contact with faculty, and higher order thinking skills.It is essential that we challenge students intellectually and help motivate them to excel in their studies.
"Task Planning", or the ability to break down tasks or projects to ensure they are completed, is linked to organizational and study skills.Goldfinch and Hughes (2007) stated that study skills are correlated with academic performance and emphasized long-term planning as a key aspect of first-year college student success.It is essential for students to break down tasks as there will be many assignments, and this level of organization will assist students in finishing their coursework by meeting stringent deadlines.
"Expectations", in this particular context, are connected to goal setting and career planning.These goals must be specific and can really improve the likelihood of success and raise students' self-efficacy.Schunk (1990) specified that goals must integrate specific performance criteria, which may enhance learning and the probability of meeting the specified goal.Goals are connected to a broad number of motivational variables that set the stage for positive achievements (Ames, 1992).Working toward a specific goal will improve students' self-efficacy, and according to Bandura and Cervone (1986), self-efficacy notably influences the amount of effort expended, and ultimately, persistence toward the goal.
"Family Involvement" is the participation and encouragement provided by family members of college students in assisting them with career planning and decision-making (2015, May 11).According to McCarron and Inkelas (2006), the involvement of parents in first-generation college students lives has a significant impact on their educational aspirations.Playing such a powerful role, it is suggested that higher education professionals working in the area of recruitment reach out to parents directly through orientation, open houses, and high school bridge programs.
"College Involvement" describes the amount of interaction that students have with their institutions.Astin (1999) provided a student involvement theory and asserts that involvement is defined as "the amount of physical or psychological energy that a student devotes to the academic experience" (p.518).The premise of his theory is that students highly involved are more likely to engage in positive activities such as studying, interacting with faculty, joining organizations, and/or spending more time on campus.Conversely, uninvolved students may not devote enough time to their studies, have less interaction with faculty, and abstain from extracurricular activities.
"Time Management" refers to college students' ability to organize and maximize their time and to rank the high priority objectives from the lower priority tasks.To be successful in college, it is important that students' manage their time wisely and efficiently.Macon, Shahani, Dipboye, and Phillips (1990) found that students who self-identified as good time managers reported greater performance, work and life satisfaction, and less somatic tensions.Not only does efficient time "Wellness" is described as the emotional and physical health of college students.In college, students are faced with a variety of obstacles and challenges, which may produce stress, sleeplessness, and inadequate diets.Pritchard and Wilson (2003) examined the impact of college students' emotional and physical health, specifically depression and fatigue on GPA and retention.It was found that students who reported they intend to drop out were more fatigued and had lower self-esteem.Conversely, students who were planning on staying enrolled developed coping mechanisms to deal with taxing situations that may arise.
"Precision" is aligned with personality type, or a particular mentality.Our mindset and personal preferences can go a long way in how we approach certain situations and face new challenges.Individuals have preferences about how they like to learn information, interact with the material, and complete tasks."Students with different type preferences tend to respond differently to different modes of instruction" (Felder, Felder, & Dietz, 2002, p. 3).It is very important that college students are cognizant of their personality types, attitudes, and preferences to ensure a proper plan of action is taken when tackling challenges related to college.
"Persistence" is linked to self-efficacy as it is defined as facing a task with selfencouragement, diligence, and personal urgency.It is how students feel about approaching their goals, objectives, and assignments.Bandura (1997) described the notion of self-efficacy as "the belief in one's capabilities to organize and execute courses of action required to produce given attainments" (p.3).Self-efficacy is having confidence in your ability to complete a particular task.Chemers, Hu, and Garcia (2001) guided a longitudinal study measuring the effects of self-efficacy and optimism on students' academic performance, which were strongly related to adjustment to the college experience.
The ten student success factors that make up the CFSI have been outlined with selected higher education research corresponding to and supporting these facets.This brief snapshot of facets stresses the qualities necessary for college students to be successful.

Methodology Purpose Statement
Pretest/post-test designs are widely used in behavioral research, primarily for the purpose of comparing groups and/or measuring change resulting from experimental treatments or interventions.The purpose of this study is to measure the gains in academic performance as measured across the ten CSFI factors of student success.The measurement of that change provides a vehicle for assessing the effectiveness of the intervention (teaching/curriculum).

Participants and Setting
The participants in the study were first-year college students enrolled in SLS1501: Strategies for College Student Success at the University of Central Florida in the summer of 2013 (N=431) and the summer 2014 (N=593).No additional demographic data were collected for the purposes of this study.The course is a 3 credit hour graded course and is 6 weeks in length.

Intervention
The CSFI was administered during the first week of class and the last week of class in each of the summer 2013 and summer 2014 course sections.Figure 1 below represents the pre/post assessment model that reflects the intervention of new curriculum in summer 2014.Recall that for the summer 2013 course, there was a uniform curricular approach to the ten CSFI college success factors.
Figure 1 below illustrates the intervention which consisted of a complete overhaul of the syllabus to include the 10 success factors that are measured in the CSFI.This means that the 10 success factors were intentionally and directly taught using a variety of instructional strategies that varied from lectures, activities, exercises, assignments, readings, and research on the topic.

Data Collection
Data were collected during the summer 2013 and summer 2014 sections of the first year experience course, SLS1501: Strategies for College Student Success.The College Success Factors Index (CSFI) was administered at the beginning and the end of each course.The instrument was completed online and the data were automatically compiled by course section and the university as a whole.The university data (all students who completed the CSFI) comprised the data analyzed in this study.
The CSFI presents college students with a series of statements that measure their ability to be successful in college by asking them to complete a Likert style self-reporting survey with five choices that range from "Strongly Agree" to "Strongly Disagree".It includes the ten CSFI college success factors previously described in the literature review.The instrument accounts for approximately ten questions centering on each of the ten college success factors, for a total of 100 items.The score range for each factor is from 10 (high success) to 50 (low success).A high score reflects negative performance and conversely, a low score reflects a high possibility of success.The lowest overall score is 100 and the highest possible score is 500.

Data Analysis
To determine if student performance improved "within" each semester independently, descriptive statistics including means and standard deviations for each of the semesters were calculated.To conclude that the differences in scores were significant, specifically that an increase in performance or "gains" existed among the summer 2013 and the summer 2014 sections, independent t-tests were conducted.

Findings
Table 1 below provides the means and standard deviations of the College Success Factors Index (CSFI) pretest and post-test at the beginning and end of the summer 2013 and summer 2014 semester course of first-time incoming freshmen.It is important to remember that the scale used for CSFI is comprised of ten success indices: Responsibility & Control; Competition; Task Planning; Expectations; Wellness; Time Management; College Involvement; Family Involvement, Precision, and Persistence, and that on this particular scale, a decrease in scores suggests an increase in student learning.These results demonstrate significant differences between the pre and post-tests of the CSFI results between summer 2013 (t-value =86.089, p<0.01) and summer 2014 (t-value =91.038, p<0.01).The means and t-values suggest the effectiveness of the curricular intervention which translates into gains in student performance.

Discussion
The steering committee that directed the administration of the course under investigation immediately realized that having students complete the CSFI pre-and post-test in the summer 2013 was a step in the right direction as collecting and analyzing data about how the students were performing in the course provided them with additional insights into student learning.Taking this a step further, the committee integrated the ten CSFI success factors into the curriculum and found that it provided significant gains in student learning.
A careful examination of the data reveals that there were improvements in performance in summer 2013 as well as summer 2014; however, there was a significant difference in the improvements made in summer 2014 than in summer 2013.The curricular intervention showed significant gains; though, there are a multitude of other considerations when "standardizing" the content occurs.Some important issues that arose were that faculty expressed concerns about unfamiliarity with teaching some of the topics, a possible loss of academic freedom (whatever that means), and a lack of autonomy in course content.This is a delicate issue when the goal is to improve the course and student performance as well as retain qualified and interested faculty from around the university.Ultimately, the steering committee found it necessary to conduct regular training sessions and hold meetings to gain the "buy-in" of the faculty.This was important because assessing improvements in student performance would be impossible without clear communication and consistent training.
The next step for this program involves continuing to improve the teaching of the success factors, the assessment of students across time, and the continued collection of data regarding student gains.Padgett and Keup (2010) suggest that what truly facilitates student development is the incorporation of vetted good practices into the course.Based on the research regarding the 10 CSFI success factors, we are well on our way to doing this.We are also progressing toward making assessment of the program part of our culture.This is a huge step considering that the 2009 National Survey of First-Year Programs (Padgett and Keup, 2011) reported that approximately one-third of higher education institutions had not conducted an assessment in the previous three years.
This research endeavor does not take into consideration the role of faculty in student achievement.The student-professor relationship is essential when considering positive outcomes for students as it impacts their quality of learning and increases the confidence that they will do well (Micari & Pazos, 2012).Pascarella and Terenzini (1979) also found that informal contact between students and professors lead to persistence and retention among first-year college students.In addition, Kuh and Hu (2001) stated that the quality of the student-faculty relationship has an effect on student learning and their overall satisfaction.In conjunction with the ten CSFI success factors, it is necessary to understand that rapport among college students and faculty is indispensable.
Additionally, the revised curriculum focused specifically on the 10 areas that were tested.As expected there was be an improvement on the test when the items have been specifically addressed in the class.In this instance, it makes sense that the course was revised based on these areas of testing.
For the purposes of this study, only the overall differences or "gains" of the CSFI were measured and reported.The aim was to find out if any gains were achieved between pre and posttest data and then again after the curricular intervention was implemented, thus laying the groundwork for future research on this topic.There is now an opportunity to analyze the ten CSFI indices individually to understand where the particular gains occurred.

Conclusion
The findings of this study suggest that the curriculum of a first year experience course is not something to be considered a "last minute thought."The way a course is taught, the topics to be covered, and the assessment of the progress of student learning must be given due diligence and adopted in a deliberate way.The results of these efforts will provide significant improvements to performance and student learning.It is highly recommended that the ten CSFI success factors are integrated into the curriculum of first year experience courses regardless of institutional type, and that data regarding student achievement is collected and analyzed to have a better understand of how to improve overall gains in learning and performance.
Cox and LemonJournal of the Scholarship of Teaching and Learning, Vol. 16, No. 3, June 2016.josotl.indiana.edumanagement help ensure academic success, it aids in relieving stress associated with poor time management and/or procrastination.

Questions 1 .
Are there gains in student performance from the pre-test and post-test in the summer 2013 and summer 2014 for the SLS1501: Strategies for Student Success course? 2. Are there significant differences in the gains in student performance from the post-tests between summer 2013 and summer 2014?Cox and Lemon Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 3, June 2016.josotl.indiana.edu

Figure 1 .
Figure 1.Pre and Post Assessment Model.This represents the curricular intervention implemented in summer 2014.
Cox and Lemon Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 3, June 2016.josotl.indiana.edu

Table 2
highlights the outcomes of the pre and post-test data of the CSFI scores between 2013 and 2014, independent t-tests were conducted and Table2provides a summary of the results.