Self-Selection Effects in Service-Learning

This study models the registration process in order to directly examine the possibility of pre-existing differences between undergraduate students who choose to take service-learning classes compared to those who do not. Foremost among the findings, on most measures there was no significant effect of selfselection. However, students who chose a hypothetical course with a servicelearning component reported higher civic responsibility and civic action scores, and reported lower socioeconomic status and greater enjoyment of past service learning than those who chose the hypothetical course without the servicelearning component. Challenges and opportunities related to random assignment within service-learning research are discussed.

The educational, civic and personal benefits of service-learning have been richly noted in an ever-expanding scholarly literature.Previous research has indicated that service-learning is related to a number of social, academic, citizenship and personal outcomes that are beneficial to students (for comprehensive reviews see Conway, Amel, &Gerwien, 2009 andEyler &Giles, 1999).For example, students engaged in service-learning show increased civic engagement, cognitive development, leadership skills, and work better in groups (Bringle, 2005;Wells & Grabert, 2004;Hollander & Meeropol, 2006).Compared to students enrolled in traditional courses, students who took part in service-learning classes showed increased grade point averages, gains in knowledge, and degree aspirations (Conway et al., 2009).Perhaps most impressive, a common theme in the literature is that the positive effects of service-learning are bi-directional; they flow from the student to the community and back to the student (Kraft, 1997).
While this research is compelling, another growing body of literature has discussed methodological flaws and limitations in service-learning research (e.g.Shumer, 2000;Bringle & Hatcher, 2000;Steinberg, Bringle, & Williams, 2010).For example, Steinberg and colleagues (2010) note several methodological challenges in designing a rigorous study of service-learning outcomes.Some of these challenges include (but are not limited to) lack of controls or comparison groups and absence of random assignment.These particular limitations leave researchers unable to infer a causal relationship between the measured outcome and participation in service-learning.Without random assignment to conditions (service-learning and at least one control group), any number of unmeasured variables could account for improved outcomes, and researchers cannot be confident that it was the service-learning experience itself that resulted in change.
For example, a study that examined reflection writings across a service-learning class found that students produced increased quality of thought, higher order reasoning skills, and improved critical thinking in their writing (Ash, Clayton, & Atkinson, 2005).However, a control group (traditional class) was not included in the study to see if these benefits are merely artifacts of frequent student writing and reflection.Without this control group, it is impossible to truly infer that the gains in writing were due to participation in service-learning (see also Bringle & Hatcher, 2000).Similarly, examining a difference at a single time-point in a class is problematic.Gallini and Moely (2003) found differences between service-learners and students not enrolled in service-learning in an end of semester survey of engagement and retention.Without a pre-test, however, limited conclusions may be drawn about this data because the two groups of students may have had pre-existing differences on measured and unmeasured variables.
Another looming confound is the opportunity for students to self-select into servicelearning or traditional classes (Steinberg et al., 2010).The issue of self-selection bias is, perhaps, the most pervasive methodological issue in the service-learning literature.In most institutions, students may simply opt-out of taking service-learning courses (unless there is a service-learning or community service learning graduation requirement).Consequently, students enrolled in service-learning courses may be different in important ways from students who are not enrolled in service-learning courses.Interestingly, many of the variables that are often reported to have changed because of service-learning are different even prior to the service-learning experience (e.g., Eyler, Giles, & Braxton, 1997;Gallini & Moely, 2003).For example, students who participate in service-learning as undergraduates are reported to have previous experience with volunteerism in their community, have high self-rated leadership ability, and to report a commitment to participating in community action programs (Sax & Astin, 1997).Essentially, students that are community and service focused self-select into service-learning courses.Thus, gains in similar measures after participating in service-learning may be indicative of the students' preference for service-learning or their receptiveness to the service-learning experience, rather than to the service-learning experience per se.This non-random assignment of participants also limits the degree to which conclusions about the efficacy of service-learning generalize from students who opt-in to service-learning to students who may opt-out or who are required to complete service-learning (see also Bringle & Hatcher, 2000).
Random assignment requires that chance alone determines whether a student is placed in a service-learning or control class.This is exceedingly difficult in educational settings where student choice is often highly valued.There is a sparse literature that attempts to randomly assign learners to traditional and service-learning sections of the same course.Typically, the sections that are service-learning are not marked for students at the time of registration, thus students theoretically select a course that contains service-learning seemingly at random (e. g.Markus, Howard, & King, 1993;Osborne, Hammerich, & Hensley, 1998).These studies typically find significant effects of service-learning on variables as diverse as self-esteem, cognitive complexity, self-concept, application of principles to novel situations, greater awareness of societal problems, grades, and learning.However, the fact that students select into particular sections at registration even without knowledge of whether or not there will be a service-learning component opens the possibility that other extraneous variables may impact these experiments with seemingly random assignment.For example, time of day, perception of assigned instructor, and even whether or not a friend is also registering for the same section may influence the outcome variables.In fact, the number of extraneous variables that could influence section choice is infinite.
Two popular ways to address these problematic methodological issues are by employing a longitudinal design, and/or by measuring and then statistically controlling for pre-existing differences between groups.In regard to longitudinal design, Astin, Vogelgesang, Ikeda, and Yee (2000) followed students over a four year period to examine change in multiple servicelearning related outcomes.Evaluating change over time strengthens conclusions about the possible impact of service on learning, but does not directly address possible self-selection effects.
Another popular way to address the possibility of self-selection is to statistically control for pre-existing differences (Astin et al., 2000).For example, if two classes had different starting scores on self-efficacy, this variable could be analyzed as a covariate allowing a researcher to compare end of term differences in self-efficacy while controlling for the pre-existing differences.Unfortunately, these statistical techniques can only control for variables that the researchers have already identified as potentially influential factors (Bringle & Hatcher, 2000;Steinberg, et al., 2010).Even statistically controlling for some variables may not fully address questions of self-selection-students that self-select into a service-learning class may show greater gains than non-service-learners because of an infinite number of unmeasured variables.Though unlikely, it is possible for any number of extraneous variables to influence outcome (e.g., degree of extroversion).While some compelling studies have attempted to statistically to control for initial differences between groups, the studies are limited in their ability to assert service-learning as the sole cause for their results because of the absence of true random assignment (e.g., Bringle & Kremer, 1993;Eyler et al., 1997;Gallini & Moely, 2003).
The goal of this research is to quantify the effects of service-learning self-selection.Because the structure of most academic settings precludes true random assignment of students to courses, we chose to model the registration process and then evaluate the effects of selfselection.In the current study, students were presented with the option of two classes that differed only by their inclusion of service-learning, and were asked to make a choice.Students who chose the course with the service-learning component were then compared to students who chose the traditional course on demographics and several commonly used outcome measures in the service-learning literature.Our hope is that this data will inform future research (and offer perspective on past research) as to the effects of service-learning self-selection.

Procedure
Participants completed all measures described below after giving informed consent.They were provided the following prompt, "Both of the classes listed below will require the same amount of in-class and outside of class work.If you were required to register for one of these courses, which would you choose?" and the following response options, "a general education class with a service learning component" or "a general education class without a service learning component."The question about class choice was always completed first, and the demographic questionnaire was always completed last, but the order of the measures described below was counterbalanced to offset possible order effects.

Measures
Demographic questionnaire.Participants were asked to report their age, year in school, sex, sexual orientation, race/ethnicity, past experiences with service learning and/or community service, and socioeconomic status.
Higher Education Service-Learning Survey (HES-LS).The HES-LS (Furco, 2000) is a 29-item questionnaire developed to assess four service-learning constructs: civic responsibility, academic attitude, career development, and empowerment.Participants responded using a 4point Likert scale, with higher scores indicating greater endorsement of the constructs.Obtained alphas for the current sample are reported in Table 1.
Civil Attitudes and Skills Questionnaire (CASQ).The CASQ (Moely et al., 2002) is a 45item questionnaire comprised of six subscales designed to assess student attitudes regarding civic action, interpersonal and problem-solving skills, political awareness, leadership skills, social justice, and diversity.Participants responded suing a 5-point Likert scale, with higher scores indicating greater endorsement of the attitudes.Obtained alphas for the current sample are reported in Table 1.Differences in leadership skills were not examined due to low reliability of this scale in the current sample.
The Self-Efficacy Scale.The Self-Efficacy Scale (Sherer et al., 1982) is a 32-item questionnaire with a Likert scale from 1 (Disagree Strongly) to 6(Strongly Agree).Self-efficacy is a person's belief that the outcome of one's behavior will be successful.Obtained alphas for the current sample are reported in Table 1.

Results
Self-selection effects were examined with independent samples t-tests.On most measures, no significant effect of self-selection was found (see Table 1).However, results indicated that course choice had a statistically significant effect on civic responsibility t(139) = 3.93, p < .01,civic action t(139) = 3.62, p < .01,socioeconomic status t(138) = -2.27,p = .02,and enjoyment of past service-learning t(72) = 4.00, p <.01.On average, students who chose the hypothetical course with the service-learning component reported higher civic responsibility (M = 2.87, SD = .33)and civic action (M = 3.70, SD = .69)than those who chose the hypothetical course without the service-learning component (M = 2.63, SD = .38,and M = 3.24, SD = .74,respectively).On average, students who chose the hypothetical course with the service-learning component reported lower socioeconomic status (M = 3.71, SD = 1.15) and greater enjoyment of past service-learning (M = 3.89, SD = 1.01) than those who chose the hypothetical course without the service-learning component (M = 4.15, SD = .95,and M = 2.86, SD = .74,respectively).

Discussion
This research modeled the registration process to examine potential differences between students who chose to engage in service-learning and students who chose traditional courses.By modeling this process, we were able to hold everything about the two courses constant with the exception of the presence or absence or service-learning, something that is rarely possible in applied academic settings.Use of this novel methodology is a significant contribution to the research literature, particularly in the scholarship of teaching and learning where random assignment is often impractical.Another notable finding of the current research was that the two groups of students shared more similarities than differences in commonly used service-learning outcome measures.This is a significant finding because it ameliorates concerns about self-selection bias.Differences, outlined below, were small in comparison to the overall finding of similarity between groups.
Despite the finding of overall similarity, students in the current study who chose servicelearning had lower socioeconomic status, higher sense of civic responsibility and action, and greater enjoyment of past service experience compared to students who chose the traditional course.In particular, it is important to note the differences in civic action and responsibility because these attributes have been reported to increase after engaging in service-learning (Moely, McFarland, Miron, Mercer, & Illustre, 2002;Moely, Furco, & Reed, 2008).The finding that students of lower socioeconomic status were more likely to choose the service-learning class is a unique contribution to the literature.Previous research has found no difference in volunteerism in college students as related to socioeconomic status (Walpole, 2003).Another study found that students of low socioeconomic status and first-generation college students spent less time volunteering than their peers (Pascarella, Pierson, Wolniak, & Terenzini, 2004).In adults, higher socioeconomic status is positively correlated with amount of volunteerism (Wilson, 2000).The current results may offer a new developmental finding regarding college students of low socioeconomic status and community service, and deserves additional attention in future research.
Although the current results are fascinating, some limitations must be noted.Our attempt to model the registration process was limited by participants' knowledge that their choice of course would not result in having to actually take the course.The choice students made in this hypothetical situation may or may not directly translate into what they would choose in a real registration process.Our sample was also primarily white and limited to one Midwestern university, and conclusions would be strengthened by replication in more diverse samples.
Applied research is inherently complicated.For example, while random assignment to service-learning is the most methodologically sound option, this is not always feasibleparticularly if other variables cannot also be controlled (i.e., instructor, course, time of offering, etc).All research designs have pros and cons, and the variety of options available allow the strengths of one design to offset the weaknesses of another.In addition to the modeling approach outlined in the current study, other applied disciplines offer potential options for creative research design (cf.clinical psychology, Kazdin, 2002).
One potentially interesting way to approach a control group is to compare students enrolled in a class with service-learning, and students who were randomly wait-listed for the same course (see Seider, Gillmor, & Rabinowicz, 2010).This option ensures that students who have similar dispositional or attitudinal characteristics based on their desire to engage in servicelearning are involved.If outcomes vary between students enrolled in the course and students on the wait-list, the additional benefits may be related to service-learning.One limitation is that this design does not rule out the fact that changes may occur from exposure to the academic material and not the service-learning itself.This design may also be problematic because of large differences in sample-sizes between students enrolled in a class and those wait-listed.In practice, this design is often limited by higher levels of attrition in the wait-listed group.This type of design, however, is another clever and innovative option for examining the effects of servicelearning on student outcomes.Some educational institutions require service-learning as a graduation requirement.This type of requirement may assist researchers in establishing appropriate control groups.If all students are mandated to complete a service-learning requirement, the wait-list control design becomes quite feasible.However, previous work has demonstrated that mandatory volunteerism has a negative impact on student outcomes (e.g.Stukas, Snyder, & Clary, 1999).Because of the differences in students who would willingly enroll in a service-learning class, instructors and researchers need to be cautious in drawing conclusions about service-learning from "mandatory" service-learning experiences.Students enrolled in a mandatory service-learning class may show null results or smaller effect sizes because the effect of service-learning may be attenuated by the presence of unwilling students.
The current study indicated that students who chose a service-learning class had higher civic responsibility and civic action, lower socioeconomic status, and greater enjoyment of past service-learning than students who chose the same class without a service-learning component.The current research also indicated that the two groups of students were similar in regard to general efficacy, social efficacy, academic attitude, career development, empowerment, skills, political awareness, leadership skills, social justice attitudes, diversity attitudes, age, and year in college.This is knowledge that has implications for how future researchers contextualize and understand their findings.For example, in the absence of random assignment and control groups, a researcher could have more confidence in results indicating higher social justice attitudes than in results indicating higher civic responsibility.
As a final thought, the methodological issues inherent in service-learning research do not undermine the value of this scholarship or this pedagogy.Our hope is to draw awareness to potential methodological issues that greatly impact the potential causal inferences that may be drawn from service-learning research.As discussed in the introduction, many methodological issues complicate the service-learning literature.In particular, this study highlights the need to consider the possible impact of self-selection, and to control for it when possible.This need is especially great when the study will examine measures that the current results suggest vary among students engaging (or not) in service-learning.Though challenges exist with random assignment within the service-learning literature, the most important finding of the current study is that students who self-select into service-learning are mostly similar to their peers who do not.

Table 1 . Differences by class choice
Notes.Skills -Interpersonal & Problem Solving Skills.Enjoy Past SL -Enjoyment of Past Service Learning.NA -not applicable.The low alpha value for the Leadership Skills scale precluded further analysis, and we caution researchers using this measure to check their sample alpha value prior to data analysis/interpretation.