Assessing What Really Matters to Student Learning Inside The National Survey of Student Engagement

ttending college is more important than ever. Social and political issues are increasingly complicated, and will become more so as the pace of change escalates. Virtually every sector of the economy l requires workers with skills and competencies beyond those most people acquire in high school. It's no surprise then that there is widespread interest in the quality of undergraduate education. State legislators, accreditors, parents, employers, and others want to know what students are learning and what they can do. George D. Kuh is Chancellors' Professor and Director of the National Survey of Student Engagement for the Center $ for Postsecondary Research and Planning at Indiana Unr5 versity Bloomington. i: p c I z e < > -

Though the NSSE survey doesn't assess student learning outcomes directly, it does provide the kind of information that every school needs in order to focus its efforts to improve the undergraduate experience.
How well are we doing? Nobody seems to know. At least that's the conclusion of Measuring Up 2000, the state report card on higher education released last fall by the National Center for Public Policy and Higher Education. The report assigned grades to each state on five of the six key performance indicators. However, in the area of student learning, all 50 states received an "Incomplete." There just wasn't enough evidence across all the states to evaluate the nature and degree of the impact of college on students. Sooner or later, colleges and universities are either going to demonstrate what students are learning or some external entity will impose its own approach.
Fortunately, there are scores of efforts underway to assess student learning and improve the quality of undergraduate education. This article describes one such initiative-the National Survey of Student Engagement (NSSE). The project's first national report (NSSE 2000: National Benchmarks of Effective Educational Practice) was released last November and the second round of data collection is nearing an end.
The NSSE project revolves around a survey of college students. But it's also intended to foster a particular way of thinking and talking about collegiate quality. As a survey NSSE annually assesses the extent to which students at hundreds of four-year colleges and universities are participating in educational practices that are strongly associated with high levels of learning and personal development. We know a lot from research about the factors responsible for high gains in learning during college (a summary of this research can be obtained on the NSSE Web site at indiana.edu/-nsse). As summarized in the classic report, "Seven Principles of Good Practice in Undergraduate Education," level of academic challenge, time on task, and participating in other educationally purposeful activities directly influence the quality of students' learning and their overall educational experience. Indices of effective educational practice can thus serve as a valuable proxy for quality in undergraduate education. Moreover, colleges and universities can take immediate action when they determine which areas of student engagement need attention. So, though the NSSE survey doesn't assess student learning outcomes directly, it does provide the kind of information that every school needs in order to focus its efforts to improve the undergraduate experience.
Equally important, NSSE is also an attempt to shift the nature of the public conversation about collegiate quality. For years we've focused on the sometimes sensationalized rankings that appear in various magazines. These rankings are based almost exclusively on an institution's resources and reputation, and say little about the student experience (see Pascarella's article in this issue). NSSE data focus on what is far more important to student learning-how students actually use the resources for learning that their school provides. This is a much different and more accurate way to think about collegiate quality than what college rankings represent. NSSE's conceptual roots go back several decades in the form of efforts to document the conditions that promote student learning (see Resources box). Among the pioneers in this effort were Nevitt Sanford, Alexander Astin, and Arthur Chickering. Chickering and Zelda Gamson later joined with a handful of other leading scholars to distill the research findings on teaching and learning at a Wingspread retreat in 1986. This group produced the "Seven Principles of Good Practice in Undergraduate Education." Several years later, in 199 1, Ernest Pascarella and Patrick Terenzini skillfully synthesized decades of research on college students that affirmed these practices and pointed to some additional conditions that enhance learning. The National Education Goals Panel in the early 1990s sparked a series of conversations about how to promote the measurement and use of good educational practices, though the political will behind the movement dissipated before any concrete action occurred.
attention every year even though they have little to do with learning. College officials try to appear disinterested (and some actually are!). But when a new set of rankings is published almost everyone scurries to find out where his or her school is rated.
In February 1998 Russ Edgerton convened a small group of educational leaders and scholars at The Pew Charitable Trusts to discuss concerns about the college rankings. Everyone agreed that alternative measures of college quality were needed, both for institutional improvement purposes and to help enlighten the public as to what really is important to collegiate quality. One of the more promising ideas was an annual assessment of the extent to which institutions were using the kinds of good educational practices identified in the literature.
Toward this end and with support from Pew, Peter Ewell convened a group of nationally known scholars on college student development with a charge to develop a short survey instrument focused on the extent to which students engage in good educational practices. By late summer 1998 the instrument was ready for field-testing and two pilot administration cycles were completed (fall 1998 and spring 1999) before the first national administration was launched in spring 2000. This work has been underwritten by a generous grant from The Pew Charitable Trusts along with institutional participation fees.
For reasons I'll explain later, the NSSE survey is administered differently than most college student surveys. An independent third party, the Indiana University Center for Survey Research, uses professional survey research techniques to send the questionnaire directly to random samples of first-year and senior students at four-year colleges and universities. Two modes of administration are used, a traditional paper questionnaire and a Web-based version. Standardized Despite all this good work, college rankings get widespread survey administration procedures guarantee that all students at participating schools have an equal chance of being selected, which makes it possible to report results with a high degree of credibility while remaining free from the direct control of either institutions or outside stakeholders.
In spring 2000, about 75,000 students at 276 schools completed the survey (63,000 made up the respondent pool used for the national benchmark analysis, the remainder represent additional students who were surveyed at the request of some institutions as part of an optional oversampling strategy). This year, NSSE is surveying more than 220,000 students from about 320 institutions. Assuming a response rate comparable to the first year, by the end of this summer we'll have information about student engagement from about 150,000 students from almost 500 different colleges and universities.
It's important that the NSSE project have a representative cross-section of institutions each year in order to establish national benchmarks. So far, the participating colleges and universities pretty well mirror all four-year institutions in terms of size. sector, Carnegie type, region, and so on. The schools come from 49 states as well as the District of Columbia and Puerto Rico. Twelve state systems have participated and 17 consortia have been formed in order to facilitate data sharing and peer comparisons-a very important feature of the project.
Co-sponsored by the Carnegie Foundation for the Advancement of Teaching and the Pew Forum on Undergraduate Learning, the first national report was designed to be accessible to audiences both inside and outside the academy. The report emphasized the important link between effective educational practices and collegiate quality by featuring five benchmarks of effective educational practice, which were created using student responses to 40 key items from the survey. The five benchmarks are: level of academic challenge. active and collaborative learning, student interactions with faculty members, enriching educational experiences, and supportive campus environment. The report also highlights some promising and disappointing aspects of student engagement. Among the former is that substantial proportions of students are getting experience with collaborative and active learning and service learning. For example, more than 90 percent worked with other students on projects during class and 63 percent of seniors reported doing community service or volunteer work. Also, most students viewed their canipus environments as supportive and responsive, perhaps ;I sign that colleges and universities are succeeding in efforts to create welcoming and affirming environments.
Less comforting is that the frequency of student-faculty interaction was much less than what research studies suggest is optimal. First-year students on average reported only occasional contact (once or twice a month) with their teachers. Seniors a1 doctoral-extensive universities had no more interaction with faculty members than first-year students at liberal arts colleges. Also, the amount of time students spend preparing for class is only about half of what is typically expected. More than half (56 percent) of all full-time students devoted only 15 hours or less preparing for class; about 10 percent spent five or fewer hours-not nearly enough, according to most faculty members, to perform at acceptable levels.
It's also interesting to note that the areas of promising performance are those that can be directly influenced by

CHANGE MAYIJUNE 2001
academic policy-things like including specific "highvalue" features of a curriculum such as requiring capstone courses, creating higher expectations, or enhancing student satisfaction with the college environment. The undergraduate reform movement that began in the mid-1980s focused on some of these features and, thus, may be having a positive impact. Harder to change, though, appear to be dayto-day behaviors like student-faculty contact and active and collaborative learning that require more fundamental changes in campus cultures.

INQUIRING MINDS WANT To KNOW
The NSSE project goals are ambitious. Once the project is fully implemented (2004) we expect to have about 1,000 colleges and universities with "fresh" student engagement data in the national database. Assuming a data "shelf life" of about four years, this means most schools would participate every third or fourth year. The stakes for institutions are higher than is customarily the case with most surveys. State systems such as Kentucky, North Carolina, and Wisconsin intend to use the information from NSSE in their performance-indicator systems-and perhaps even in funding. Prospective students might use the results in deciding which colleges to apply to and, ultimately, to attend.
In the course of designing and implementing NSSE, we encountered a number of vexing issues and dilemmas. Resolving them required some specific (and occasionally controversial) decisions about how to proceed. on student testimony. The survey questions have substantial face validity and many have been used for years in other college student surveys. Even so, some faculty members and administrators wonder whether they can trust what students say about their experiences. that self-reported information is likely to be valid if certain conditions are met, and the The College Student Report was designed accordingly. The survey questions are clearly worded and refer to recent activities with which students have firsthand experience. The questions don't intrude into private matters nor do they prompt socially desirable responses. Psychometric analyses produce acceptable levels of reliability and demonstrate reasonable response distributions for most items. It's also the case that student reports about certain matters are the only feasible, cost-effective source of this kind of information. For example, it would be prohibitively expensive (and probably logistically impossible) to observe directly how students at large numbers of institutions use their time and the extent to which they interact with peers and faculty members.
responses to the questions about the gains they've made in various areas during college. This is valuable information for individual institutions but these data can't reliably be used to represent or compare student learning outcomes, either at the national level or between different types of institutions. Students start college with different levels of knowledge and competence and there is no way to responsibly take these differences into account when interpreting and comparing their self-reports of how much they have gained.

Can Students Be Trusted? NSSE, of course, relies entirely
A considerable body of social science research documents One area where we've got to be especially careful is student Why Use (These) Benchmarks? To accomplish the purposes of the project, answers to many survey questions must be. made understandable to a wide range of interested parties. To do so, we had to reduce the more than 60 questions on the NSSE survey to a handful of self-evident concepts. After analyzing the data in different ways we ultimately decided to create five benchmarks based on 40 items. The resulting benchmarks serve three important purposes.
First, they represent educational practices that resonate well with faculty members and administrators. They also are understandable to people outside the academy like parents of prospective students, accreditors, and so on. Second, the benchmarks empirically establish current levels of student engagement in effective educational practices nationally. As such, they represent a baseline against which future performance can be compared. Thus we can monitor progress over time, similar to what is intended in such K-12 efforts as the National Education Goals Panel and publications like Quality Counts.
across different sectors and types of institutions. Results from NSSE's first year suggest that though small colleges generally outperform large universities, there is considerable variation within all types of institutions. That is, there are some larger universities where students are more engaged than their counterparts at some small colleges.
Why Name "Names?"NSSE is intended to get people thinking and talking productively about collegiate quality. At the same time it must be a reliable tool for institutional selfanalysis and improvement. Balancing these two objectives is tricky. With advice from the NSSE National Advisory Board, we developed a participation agreement that allows the project to use pooled data to establish national benchmarks and permits public disclosure of institution-level results, but only with the explicit consent of the institution. These disclosure policies were not part of the original design but were put in place prior to the first national administration because of substantial (and understandable) institutional resistance to the prospect of universal public disclosure.
We named a total of 49 institutions in the first national report because of strong or noteworthy performance in a particular area. (We could easily have named another 50 or more colleges and universities had we identified the top performers in each of the Carnegie types on all five of the benchmarks). Only one school eligible for mention declined the opportunity to be named. Media attention to some of them-Beloit College, Elon College, Sweet Briar College, and the University Third, benchmarks allow us to compare student performance of Virginia-was considerable and, by all accounts, quite beneficial. At the same time, a few people grumbled that this approach might "force" their schools to disclose results too. But, we haven't seen much of this, nor has there been any identifiable backlash in participation thus far.
Clearly, policies governing the use of institution-level results in the coming years are of paramount importance. Some institutions will likely make their results public, especially if they are exemplary. Similarly, state system participants may publish NSSE results on a comparative basis in order to respond to growing accountability demands. Our understanding is that sunshine laws in many states would require the disclosure of NSSE results for all public institutions, should anyone inquire. Finally, extant (but largely untested) freedom of information laws may ultimately compel institutions to disclose their results in spite of any established NSSE policies. More than a few public and private colleges and universities have put their results on their Web page. tive for NSSE's inaugural administration was to document the current context of student engagement at four-year colleges and universities on a national basis and establish a baseline against which future performance could be judged. For this reason we made only a few statistical adjustments when computing institutional benchmark scores. For example, to adjust for response bias at the institutional level, we made the responses of those who filled out the survey look like the overall profile of students at each institution. We also adjusted the responses of full-time and part-time students for the four items (reading, writing, amount of time spent studying) where students answers will depend directly on the number of classes they are taking.
One could argue that individual schools might have fared better if certain variables were held constant when calculating the benchmarks, like educational expenditures per student and student ability. We explored this, and did decide to provide schools with information that essentially shows what their students would be predicted to do in terms of effective educational practices (given their backgrounds and selected institutional characteristics) and what they actually reported doing. But, we didn't publicly report these results for two reasons. First, the analytical approach itself needs more testing and we need more complete information about certain student and institutional characteristics. Second, as mentioned earlier, the terms of institutional participation do not at this point allow us to report individual institutional-level results. what they are doing to enhance the quality of the undergraduate experience. At this juncture the promising signs far outnumber any undesirable project outcomes. But there are some potentially sensitive matters that must be managed appropriately in order to attain NSSE's long-term goals. In this section we look back on some key events and the issues they raise.

Why Not
Promoting "The Good" What many participating schools find attractive about NSSE is getting high quality information about key aspects of the student experience. Without knowing how students spend their time, for example, it's almost impossible to link student learning outcomes to the educational activities and processes associated with them. In the absence of this information it's hard to know where to target institutional effort and resources in order to enhance student learning. In more than a few instances, data from the NSSE project are the only reliable source of information that a school has about student engagement in effective educational practices. Most institutions don't routinely collect this kind of data. Those that do like the fact that NSSE data are comparable across different types of institutions.
In one sense, NSSE is like "institutional research in a box" because all participating institutions have to do is provide an electronic data file with student contact information and institutional letterhead. The NSSE project team does the rest. Each school gets a customized institutional report, including some preliminary data analysis. NSSE also places the information in a context in which a school can readily compare its performance against its peers. NSSE provides benchmarks for first-year and senior students for various types of institutions. This allows schools to readily identify areas where their students are performing above or below the baseline typical of schools like them. Many schools are leveraging local use of the information by comparing their results with those from peer institutions, sometimes as part of a consortium arrangement created either a prior; or after the data have been collected. Comparing student engagement information against institutions with similar missions and student characteristics adds legitimacy, and often a sense of urgency, to institutional improvement efforts.
Also, schools get their own data so they can further analyze their results. In some settings it's enough to provide information that highlights strong or weak areas of student performance. But without additional fine-grain corroborating data, a school may not be able to convince faculty or staff that changes in pedagogy or policy are warranted. Faculty members in particular are eager to see data disaggregated by major field. Thus, the greatest impact and utility of NSSE data will come when they are integrated with other institutional data about the student experience. For example, the director of institutional research at one private college is linking NSSE results with student records including results from the freshmen CIRP questionnaire administered by UCLA and other

C H A N G E M A Y l J U N E 2001
surveys to look at the kinds of experiences with faculty members and peers that contribute to various outcomes of college. To increase analytical power, the college opted to use the NSSE oversample option so that 50 percent of its students would be surveyed including sophomores and juniors (classes that are not part of the standard NSSE sampling scheme).
We periodically invite participating institutions to share with us the ways they are using NSSE data. So far. their response suggests that NSSE is well received and its results are beginning to be used at both the individual campus level and in some state performance reviews. Up to this point the most common use of NSSE data is the circulation of the project-generated reports augmented by additional institutional analysis either conducted locally or customized analyses performed by NSSE staff. Internal groups that have discussed NSSE data include: the president's cabinet; the provost or chief academic administrator, academic policy committees, strategic planning committees; vice president for student life, members of the student affairs staff; division of enrollment management, financial aid, admissions, and retention committees; the faculty senate and other faculty groups or departments; media representatives (ostensibly for marketing purposes); and student groups.
According to Trudy Banta, Indiana University Purdue University Indianapolis intends to use NSSE data to monitor progress on three of the institution's six Principles for Undergraduate Learning: 1) communication skills-by looking at student responses to perceived gains in items related to writing, speaking, and using technology; 2 ) critical thinking-by looking at perceived gains in thinking critically and analytically and at opinions about the emphasis in courses on applying, analyzing, synthesizing, and making judgments; and 3) understanding of society and culture-by looking at responses on the NSSE items about conversations with other students with different beliefs and values or of different races or ethnicities, whether the institution encourages contact among students from different backgrounds, and students' perceived gains in understanding people of other racial and ethnic backgrounds. NSSE data may also be used to document the extent of students' civic engagement (such as, required participation in a community project or voluntary community service, perceived gains in the likelihood of voting in elections or contributing to the welfare of their communities).
Georgia Tech University created a $250,000 fund to support faculty research that involved undergraduates in response to NSSE data showing that first-year students were not working with faculty members on research as extensively as the institution thinks desirable.
NSSE data are also being used for inter-institutional collaboration. For  One of NSSE's continuing challenges is attaining student response rates that are high enough for institutions to be confident that the results are valid and stable.
to learn more about the student experience at their respective schools and others like them.
It's possible that NSSE data might in the future be incorporated into one or more college guides and be used by high school counselors. Up until now the most frequently mentioned external groups with which institutions expect to share their NSSE results are national accreditation agencies and professional accrediting groups, state higher education commissions, foundations, prospective students and their parents, and alumni. Longwood College is incorporating NSSE results into the institution-specific performance measures that it's required to report to the Virginia State Council of Higher Education. In addition to the five benchmarks of effective educational practice, Ed Smith, Longwood's director of Institutional Research, is planning to create indicators from combinations of NSSE items that represent development of work-related knowledge and skills, participation in co-curricular activities, civic virtue, and technology use.
Finally, though the NSSE project addresses only the fouryear college sector, the intention was always to extend the good practices paradigm to the two-year sector once the instrument and survey process were established. A version of the NSSE survey is now being developed for two-year colleges by a team at the University of Texas at Austin. The Community College Survey of Student Engagement (CCSSE) is supported by matching grants from The Pew Trusts and Lumina Foundation. It's likely that the addition of CCSSE will leverage even more interest on the part of state systems in using the four-year version, because the CCSSE will now make it possible to compare student engagement across these two major sectors of postsecondary education.
Addressing the "Bud" Any large, complex project will hit some bumps along the way. One of NSSE's continuing challenges is attaining student response rates that are high enough for institutions to be confident that the results are valid and stable. So far the response rate has been about 42 percent for each of the three administrations (the two field-tests in 1999 and the spring 2000 national administration). While this is higher than many institutions get with their own surveys (even those that offer cash or other incentives), some schools do better (especially when the survey is tied to registration or graduation). We continue to make modifications in the questionnaire and administration process to try to improve response rates. For example. with the help of survey expert Don Dillman we redesigned the survey for 2001 so it looks less like a "test" and is easier to complete. We also intend to introduce experimental items from year to year as issues emerge that are of interest to institutions and the public. Surveys are an important vehicle for understanding the campus environment for learning, which explains why college students get bombarded with questionnaires. But so many surveys floating around a campus dampens response rates in general. NSSE uses technology whenever appropriate (for example, Web survey and e-mail reminders) to cut down on the paper flow into student post office boxes. We also have the capability of working with institutions to draw samples in a way that ensures that students don't receive multiple surveys when two or more different instruments are being administered. To ensure the highest quality information, NSSE is committed to working closely with individual campuses and other national research programs to address this legitimate concern in a responsible, coordinated way.
An unanticipated side effect of NSSE is that at some institutions, more people were involved in deciding which student surveys to use. Choosing assessment tools was once the exclusive province of the Institutional Research (IR) office. But, to accomplish the purposes of the NSSE project it is important that presidents and institutional public affairs personnel are involved and informed. For this reason we send annual invitations to participate to presidents, senior academic officers, and institutional research office directors at all four-year colleges and universities. At various points during the survey administration cycle we communicate directly with the president's office in addition to the designated institutional contact (who is typically an institutional research officer). As the first national report release date drew near we also wanted various officials at the NSSE 2000 institutions to have time to digest their results and to determine how to explain the meaning and importance of student engagement in effective educational practices in their particular context. We again provided them with relevant information and an opportunity to ask questions.
In a few instances participation in NSSE 2000 or 2001 was added to the IR office's pre-determined schedule for administering other surveys. In an era of increased accountability, NSSE-like tools may require that institutional research officers rethink their role in marshalling and interpreting assessment data. What appears to be a simple internal coordination issue may be a harbinger of a shift from insular decision-making by IR professionals to a more collaborative model for deciding what types of assessment tools to use. This prospect could actually elevate the visibility and influence of the IR function if NSSE and related data are used more frequently to guide improvement efforts.
Finally, NSSE is not the only good instrument out there for assessing the experiences of college students. Other prominent tools regularly used by many colleges that have excellent grounding in the research literature include the Cooperative (Continued on page 66) Avoiding the "Ugly" In principle, the NSSE project is positioned to benefit all parties-participating institutions, external authorities, prospective and current students, and the public. But the potential also exists for NSSE data to be used in unintended, untoward ways. Institutions have insatiable appetites for positive media coverage. Performing well on the NSSE is thus a potential media event. So far participating schools and the media have handled this with admirable discretion. Press coverage generally has been responsible, and has captured the right toneemphasizing that the NSSE is a tool to estimate collegiate quality that stops well short of identifying "winners" and "losers" in the game of undergraduate education. But one can imagine an institution crossing the line by holding press conferences, putting up billboards, and distributing viewbooks claiming that Siwash University. is "the most engaging college in the region bordered by three states and an ocean" (or mountain, or major river, etc.).
NSSE could get really ugly if schools were to begin competing for students or foundation support on the basis of NSSE results or if some irresponsible third party created rankings from the limited information that is publically available in the national report or from participating schools. Both these scenarios are at odds with the project's goals, guiding philosophy, and policies.
Another potentially ugly outcome would be for the media or politicians to conclude that there is a single "best" form of undergraduate education and propose a "one size fits all" solution to improving the baccalaureate experience. Based on the first national report, it seems that the residential liberal arts college is superior in many respects. But underneath benchmark scores that favor many (though clearly not all) small residential colleges is the notion that collegiate quality has multiple dimensions. NSSE data offer the promise of develop-

66
ing new ways of looking at colleges based on effective educational practices. Familiar ways of categorizing colleges, like the Carnegie Classification of Institutions, are not very revealing when it comes to differentiating quality in the undergraduate experience. In the national report we experimented with several approaches to identify "intellectually challenging," "writing intensive," and "civic-minded" institutions. This resulted in grouping some selective and general liberal arts colleges with liberal-learning public regional institutionsand occasionally some large universities. We can also avoid cookie-cutter judgments about collegiate quality by identifying "Colleges that Beat the Odds." One way to do this is to compare students' expected and actual performance on the NSSE benchmarks. This approach could parallel recent efforts to report retention and graduation rates only after taking into account a number of important institutional characteristics. Using a regression approach, an individual school's performance would be estimated after controlling for a range of factors like selectivity, size, and program mix that are related to outcomes. This way an institution can compare its students' actual performance with what might be expected. "Best practices" associated with schools that perform better than expected might then be highlighted and, again, the presentation might be most effective by identifying exemplar institutions. The relative feasibility of this approach needs to be explored through further data analysis. In addition, such matters as the degree to which a "net effects" argument is understandable to the public would also need to be explored.
judgments about collegiate quality until more institutions are in the database and until we have drilled down further into the data. Fine-grained information is needed in order to gain a deeper understanding of collegiate quality, to guide institutional improvement efforts, and to identify the kinds of questions that prospective students should ask in their search for a "good college for them." We also need to be vigilant in helping accreditors and state and governmental officials use NSSE results responsibly to estimate those aspects of the undergraduate experience that contribute to student learning.
All this is to say that it is premature to make far-reaching A FINAL WORD NSSE is but one of many initiatives undertaken to respond to increasing demands for evidence of collegiate quality. Its longterm success depends on the continued cooperation and commitment of forward-looking leaders like the presidents, deans, and institutional research officers who stepped out in front to involve their schools in NSSE. It also depends on the responsible use of the resulting data by accreditors, state system officials, and media representatives who understand the need to make the public more aware of what really matters to student learning.
Together, we've navigated some uncharted waters in launching NSSE and in trying to demonstrate the project's utility to constituents inside and outside the academy. The reception from all quarters has been positive, suggesting that the time has come for NSSE-like tools and data. More important, the project's early success bodes well for enhancing college quality because NSSE results, in combination with other sources of information, can point colleges and universities to the specific areas of educational practice where improvements related to learning can be made. @