Cold Calling and Web Postings: Do They Improve Students’ Preparation and Learning?

Getting students to prepare well for class is a common challenge faced by instructors all over the world. This study investigates the effects that two frequently used techniques to increase student preparation -- web postings and cold calling -- have on student outcomes. The study is based on two experiments and a qualitative study conducted in a statistics course that Masters in Public Policy (MPP) students take in their second semester at the Kennedy School. When used together, web postings and cold calling seem to increase the amount of time that students devote to reading before class by about an hour. This effect is both statistically and practically significant. However this increase in pre-class reading did not translate into increased learning (measured by average test scores on the midterm exam). Neither of the two techniques seems to be better than the other one at increasing reading time, test scores or other student outcomes.


Introduction
Getting undergraduate and graduate students to read for class is often difficult (i.e. McKeachie and Svinicki, 2010). Students have significant demands on their time, including other schoolwork, extracurricular activities, and socializing. Examining undergraduate and graduate-level psychology courses, Burchfield and Sappington (2000) found that student completion of assigned readings plummeted from approximately 80 percent in 1981 to 20 percent in 1997. As an additional hurdle, statistics reading is often found by students to be dry and hard to follow (i.e. Cobb 1987). In an examination of introductory statistics textbooks for higher education, Harwell et al (1996) found that only one of the six texts was recommended for use by a majority of students.
Even if students do the readings, there is no guarantee that the experience will make a significant impact on the student's performance in the course. Creating usable knowledge from reading involves a set of complex skills, and these skills are rarely taught to students in an explicit fashion (Nilson, 2010). Research has also documented the difficulty students have in transferring knowledge learned in a particular context to new problems and settings (Bransford et al, 2000). This difficulty can be particularly acute with textbooks, as these resources often present material in ways that are quite different from the style and substance of the rest of the course.
There are several ways that instructors can encourage students to increase the quantity and quality of their reading. Two promising techniques, both commonly used at the graduate level, are web postings and cold calling. This paper reports on a study conducted to find out how effective these methods were in improving student outcomes in a masters level statistics course taught at the Kennedy School of Government (Harvard University).
The rest of the paper is organized as follows: Section 2 provides a description of the two techniques and explores the mechanisms through which these techniques could potentially impact student outcomes, Section 3 describes the study design, Section 4 presents the results, and Section 5 concludes.

Description of Web Postings and Cold Calling
This section provides a brief description of the two teaching techniques under study--web postings and cold calling. Although each technique can be used in a variety of ways, particularly in terms of the types and number of questions employed, this background section will focus on a robust format of each technique.
Web postings use modern technology --particularly the course management platforms common in most institutions of higher education --to facilitate a long-standing practice of having students answer questions about the assigned reading in advance of coming to class (Chance et al, 2007). Peer instruction (PI), a popular pedagogical method in higher education, utilizes a well-developed and representative model of web postings (Crouch and Mazur, 2001). In PI, web posts are oriented towards higher-order cognition, asking students to carry out tasks including application, analysis, and evaluation. The assignments also ask students what they have found most confusing in the readings. The students post their answers through the course site in advance of class, giving the instructor the opportunity to read the responses before class. This information can then be incorporated in to the planning and execution of the class session. The students' posts, and their related participation in class, are graded to incentivize compliance.
Cold calling, although most well known for its use in law and business school classrooms, is also common in many other higher education settings. Cold calling is the process of asking a question and calling, without advanced notice, on a student who has not volunteered to answer that question. Often, instructors devoted to cold calling will begin a class session by volunteering a student to summarize the essential aspects of one or more of the readings (Garvin, 2003). Cold calls are then often used significantly throughout the remainder of the class session, employing questions that range from those that elicit factual details to those requiring complex thinking skills. Often, an instructor will stay with a single student for a long string of questions, and in the most extreme form, will do so until the student comes to a question for which they have no satisfactory answer.

Theory of Change: Potential for Impacting Student Learning
This sub-section describes the mechanisms through which these techniques could possibly affect student learning. Below is a simplified logic model encompassing the theories of change for both pedagogical techniques  The remainder of this section expands on each of the four mechanisms described in Figure 1, and also explores a few mechanisms through which the two techniques could lead to a decrease in student learning. a) Increased learning due to the task itself Web postings are one version of what the learning sciences refer to as "prompts." There is a rich research literature on how a wide variety of prompts increase student learning (e.g., Chi et al., 1994;Linn & Eylon, 2006;White and Frederiksen, 1998;McKeachie et al., 1990). Prompts are a type of scaffold: a pedagogical tool designed to assist learners in accomplishing complex tasks more effectively (Wood et al., 1976). Reiser (2004) distinguishes two main ways in which scaffolding supports learning: "structuring the task" and "problematizing the subject matter." "Problematizing the subject matter" happens when a scaffold makes some aspect of the student's work more problematic, thereby requiring the student to spend more time and mental energy addressing this important idea or process. These "desirable difficulties" have been shown to improve student's conceptual knowledge (Linn, 2006). Web posting questions serve this role by requiring students to grapple much more intensely with important concepts in the readings. The answers to these questions also make visible, to both the student and the instructor, the student's current ways of thinking about the material. Research has shown that engaging student preconceptions in this way is critically important to the development of lasting knowledge and skills (Bransford, et al, 2000).
The web post question that asks about remaining areas of confusion also provides valuable information to both the student and instructor. Answering the question helps the student pay more attention to the process of learning -what is going well, what is still problematic, and what the student might do to go about addressing these problems. This "thinking about thinking," or metacognition, helps students become better self-directed learners and problem solvers (Bransford et al, 2000). The instructor, in turn, can use this information to help him/her figure out how best to spend class time.
Cold calling facilitates students' abilities both to process information quickly and to communicate a response effectively. There are several ways students benefit from this process. First, when people are forced to verbalize their understanding and make connections between ideas, the knowledge involved is strengthened (Linn, 2006). Second, as with web posts, cold calling helps make thinking visible. In particular, due to the involuntary nature of cold calling, students' misconceptions and preconceptions are more likely to surface than if instructors solicited volunteers. Although this can be uncomfortable, unearthing these misunderstandings is an important step in helping students to learn concepts effectively (Bransford et al, 2000). Last, research has shown that cold calling can lead to greater participation of minority groups; for example, US college classes featuring only voluntary participation tend to be dominated by white males (Nilson, 2010). b) Increased learning due to the task's impact on the amount and timing of reading Extensive research has shown that holding students accountable for assigned reading in some concrete way dramatically increases the average amount students read (Nilson, 2010). Mazur (2000) has found web postings to be an effective way to get students to complete the readings, and Carney et al (2008) found similar results from using a paper-based version of the method. Although we were unable to uncover any studies of how cold calling impacted student reading completion, it is sensible to expect that students will read more given that they may be asked to answer questions in front of their classmates and that their answers are formally evaluated. In turn, increased reading could likely have a positive impact on student learning, both because reading is generally an effective way to learn material (McKeachie & Svinicki, 2010) and because research shows that a student needs to encounter information multiple times in multiple ways to effectively incorporate it into their existing knowledge (Nuthall, 1999). c) Increased learning due to the task's impact on the quality of reading Web posts are an effective method for helping students to "structure the task" of reading a textbook by drawing students' attention to important aspects of the text and by encouraging them to process the concepts in a deeper way. Even though these questions are not intended to cover every major concept of a reading, they give students a clearer idea of the types of information they should focus on acquiring. It is likely that this improved understanding of where to focus their time and attention while reading will increase their conceptual understanding. Under related conditions, Reder and Anderson (1982) found that students who read textbook summaries actually acquired more knowledge than those who read the entire text. The questions also incentivize students to digest and process the text more deeply, which research has shown leads to improved knowledge development (Ferguson-Hessler and de Jong, 1990;McKeachie and Svinicki, 2010). Students also tend to retain information more clearly and permanently when they space out their readings rather than cram them before an exam or paper (Reder and Anderson, 1982).
The influence of cold calling on the quality of reading is more indirect, as it is not tailored to any particular assignment. The types of questions that are asked in the process of cold calling will serve as a powerful signal to students as to what is valued from the reading (Doyle, 1983;McKeachie and Svinicki, 2010). If the instructor commonly cold calls on students to give a summary of one or more readings, people are more likely to read with this end goal in mind. Similarly, if the bulk of the cold calls focus on higher-order cognition, students should tend focus more on concepts than factual details. Also, as with web postings, students should benefit from cold calling if it leads them to do their readings regularly rather than all at once before an exam. d) Increased learning due to the task's impact on the classroom experience In having some positive impact on the quantity and quality of reading completed by students in advance of class, both web posts and cold calling enhance students' ability to learn from the class session. Students are better able to comprehend what is happening in class and are more aware of the knowledge gaps they need to remedy.
Both of these reasons should increase the likelihood that students are willing and able to participate in class, and this active processing of material promotes learning (Bransford, et al, 2000).
Another way that web posts improve the classroom experience is in helping the instructor make better use of class time. By reading the posts, the instructor has a fuller understanding of where students are at in terms of conceptual understanding of the material. The instructor can weave specific posts or themes from the posts in to the session and can modify what content is covered and the ways in which it is presented (Mazur, 2000). A particular advantage of cold calling, assuming it is done in a sustained fashion, is that it creates a concrete incentive for students to stay focused throughout an entire class session. Increasing student attention is important, particularly given the findings that students have a difficult time focusing on a lecture for more than ten to twenty minutes (Gardiner, 1994).

e) Decreased learning concerns
There are two main reasons why either of these methods might decrease student learning. The first involves the potential these methods have to negatively impact students' intrinsic motivation for the course. Hobson (2002) found that intrinsic motivation was significantly influenced both by the instructor's attitudes and behaviors and by the relevance of the course content. There is a danger that web posts or cold calling, particularly if mainly instituted to increase student reading, can be seen as paternalistic. Some students also find cold calling off-putting due to its adversarial nature of forcing participation (Williams, 1992). Also, motivation could be further diminished if students find the reading to be dry and difficult to understand.
The second reason why learning may decrease is that the increased time spent on reading and / or web posting may lead to a decrease in time spent on other, possibly more valuable, aspects of the course. If students are strongly incentivized to spend more time on reading and related activities such as web posting, that time has to come from somewhere. If at least some of that time is taken from other course activities -such as thoroughly understanding a problem set or studying lecture notes -any gains from the reading and related activities could be diminished or completely erased. In a detailed study of pedagogical innovation in engineering classrooms at MIT and the University of Cambridge, researchers found this ripple effect in time allocation, as a change in one aspect of the course led students to change the time they devoted to many other aspects of the course (Seering and Britter, 2007).

Study Design
Two experiments were conducted to assess the effectiveness of cold calling and web posting in increasing students' preparation for class and learning. Each experiment took place in a different section of "Quantitative Analysis and Empirical Methods II" (coded API-202), a required course taken by MPP students in their second semester. It is the second course in the statistics sequence and focuses on applied statistical analysis. Both sections were taught by one of the authors (Levy) in the Spring of 2009. The readings in the course came from "Introduction to Econometrics" by Stock and Watson, a standard statistics textbook used across many university campuses.
The two experiments followed the same basic structure (Figure 2). Levy announced the experiments in the second class of the semester, after enrollment settled in and students could no longer switch to other sections of the course. In the third class, students in each section were randomly assigned into two different groups. The students were then exposed to their respective treatment conditions in classes #4 -#11, and the midterm exam occurred in class #12. Although the experiment concluded at that point, the assignment to the two groups was reversed so that all students were treated equally in the course (i.e. they were exposed to the same interventions but at different points in the semester).

Figure 2 -Structure of Experiments
The two experiments differed in terms of the treatment conditions students were assigned to (Table 1). In experiment #1, students in the first group were assigned to the "Web Posting AND Cold Calling" treatment, which meant that they would be required to do web postings before each class and they were put on a list from which they could be cold called. Students in the second group were instead assigned to a control condition in which Levy simply encouraged them to read before class, as he had done in many of his past courses (with some exhortations about the importance of reading before class for their learning). In experiment #2, students in the first group were required to do web posting for each class, whereas students in the second group were subject to cold calling. In addition to the experiments, a qualitative study was conducted aimed at understanding students' views on how they were experiencing the web postings and cold calling. Throughout the semester, Levy met with small groups of students to ask about their perceptions, and at the end of the course students were asked to fill in a brief anonymous survey in which they indicated their predictions as to which treatment would work and why. The qualitative survey was instrumental in understanding the results of the experiments and in helping Levy draw lessons for his pedagogy.

Implementation of the Treatments
This sub-section describes in some detail how the web postings and cold calling were implemented. We think this specificity will help readers interpret the results and assess the extent to which the findings are generalizable to contexts that are relevant for them.
Web postings: Students were required to post answers to three questions on the course website by 4:00 am of the day of class. The three questions were based on the reading assigned for that day's class. Students could see each other's postings, but the questions were crafted such that it would be very evident if one student simply copied from another. For example, in the first question used in class #6 (see Box 1 below), students were asked to come up with a bivariate relationship in which they were interested and then use that bivariate relationship to answer some sub-questions. Since there are countless of bivariate relationships they could come up with, it would be very unlikely that two students would come up with the same answers unless they had seen each other's work.
The third question was always the same: "Please tell us what you found difficult or confusing in this reading assignment." This question, recommended by Harvard Physics Professor Eric Mazur, was meant to facilitate metacognitive thinking from students and to give the instructor a sense of common student difficulties. Levy used this information to adjust the length of class time spent on each topic, and at the start of each class, he shared with students the themes that emerged from the posts.
Compliance with web postings was fairly high. On average, students submitted 7.07 postings out of the 8 that were assigned. Approximately 80% of students posted 7 or 8 times. Some postings clearly demonstrated careful reading and reflection, whereas others seemed much less thoughtful. An example of the latter, in response to question #3, was: "Homoskedasticity and heteroskedasticity are interesting, but it would be great to spend some more time talking about both concepts in class." Cold Calling: As described in Section 2, implementation of cold calling can vary tremendously from class to class. In the case of this study, for each class session, Levy randomly chose one student from the cold call list of approximately 40 students. He typically asked that student two to three related questions. The questions, based on the assigned reading, tended to be factual/technical in nature. For example, in one class he asked "What are the two conditions for Omitted Variable Bias?" A student who had done the assigned reading should be able to answer this question, whereas a student who did not do the reading would have difficulty saying anything meaningful (unless they were familiar with the material from prior coursework, which was rare). When a student did not know the answer Levy would alternate between silence and probing for about 2-3 minutes, and then would move on. Based on qualitative feedback, some students felt stressful about being cold called whereas others felt that the probability of being cold called was fairly low (since only one out of the approximately 40 students on the cold call list was cold-called per class). We would rate this implementation of cold calling as "low to moderate" intensity, in contrast to the higher intensity typical of the Harvard Business and Law Schools, where instructors often do numerous cold calls per class and can probe with the struggling student for 5 to 10 minutes.

Implementation of Random Assignment
Random assignment was conducted separately in the two sections of the course. There were 80 students in experiment #1 and 93 students in experiment #2.The instructor and grade obtained in the first statistics course of the sequence were used as strata for random assignment. As it turns out, the grade in the previous statistics

Box 1 -Example of Web posting questions (Class #6)
(1) Think about a bivariate relationship you are interested in. course was highly predictive of performance in the course in which the experiments were conducted. 1 This stratification increased the statistical power of the study design, which was especially important given the relatively small sample sizes.
The goal of random assignment was to ensure that the two groups in each experiment were identical at the outset of the study, so that any subsequent difference in outcomes could be attributed solely to the difference in treatments received. Tables 2a and 2b compare the two groups for each of the experiments. As can be seen, the groups are indeed comparable to each other in key baseline characteristics, including gender, classroom seat location, previous statistics course (API-201) grade 2 , the percentage taking the advanced section of the previous statistics course (API-201A), and the grade in problem set #1 (which was due before treatment began). The differences between the groups tend to be small and not statistically significant. In experiment #2, one difference (for whether the student was an MPP1 only, as opposed to a student from another school or doing a joint degree with the Kennedy School) was statistically significant at the 10% significance level. All in all, the results of Tables 2a   and 2b are consistent with random assignment being well implemented and generating two comparable groups. The R 2 of a regression of midterm exam score in the course studied here on the strata dummy variables was 0.55 for experiment #1 and 0.57 for experiment #2. 2 This finding is important, but is to be expected given that this variable was used as strata for random assignment. 3 p-value reported to assess whether the difference between the means of the groups was statistically significant. 4 pp means percentage points.

Impact Estimation
Impacts were estimated separately for each of the experiments using the specification below: where Y is the outcome of interest, TREAT is the binary indicator for whether the student was assigned to group #1, and STRAT is a collection of binary indicators representing the different strata that were used for random assignment. The coefficient on TREAT represents the impact estimate of the group #1 treatment relative to the group #2 treatment. Robust standard errors were calculated.
The key outcomes examined were the following: (1) Time reading the textbook (minutes per week): Refers to the number of minutes students spent on course-related activities on week #4 of the semester. Students were asked to report this measure on an assignment they turned in that week. 5 (2) Time spent on the course (minutes per week): Refers to the number of minutes students spent on courserelated activities on week #4 of the semester. It was taken from the same source as the previous measure.
(3) Number of times participating in class: A course assistant reported the number of times students participated in each of the classes during the treatment period.
(4) Problem set grades: Average grade of the 3 problem sets that were due during the treatment period (on a scale of 0 to 3).
(5) Standardized midterm score: Midterm exam score was standardized by subtracting the mean and dividing over the standard deviation of midterm score in each of the two samples.

Results
Overall, the key results of the two experiments were: (1) Web Postings and Cold Calling had a positive effect on the amount of time students read before class but not on academic performance (as measured by the midterm exam score), (2) When tested against each other, neither of the two methods (Web Posting and Cold Calling) came out on top in terms of improving any of the key outcomes. This section presents results from the two experiments in more detail. 5 While these self-reported measures are subject to measurement error, we tried to minimize the degree to which this error might be systematic by including a note on the assignment indicating "Your answers to the questions in this sheet will help us understand better how you allocate the time you spend on this course. Please answer the questions candidly. Your answers will not affect your grade on this problem set or on any other part of the course. As long as you submit your answers, you will get full credit on this part of the assignment."

Experiment #1: Web Postings and Cold Calling vs. Control Group
Web Postings and Cold Calling (WP&CC) together increased the time students spent reading the textbook by approximately one hour per week (Table 3a). This effect is statistically significant, and given that the control group only read about 35 minutes per week, represents quite a large change. The effect on reading time translated into an increase in the overall time that students spent on the course outside the classroom as a result of being assigned to the WP&CC group. The magnitude of the effect on total time spent on the course is about 47 minutes, which though not statistically different from the effect on reading time, hints at the possibility that reading time may be crowding out other course-related activities. Since both time spent on reading and time spent on the course was self-reported, one could suspect that the results presented above may overstate the true effect of the WP&CC treatment since students who were assigned to this treatment may be particularly prone to overstate the amount of time they devoted to the course for social desirability bias and similar reasons. While this possibility cannot be discarded, we do not think it is driving the results observed for three reasons. First, as reported in the previous section, the students were explicitly told that their answers about time use would not affect their grades, and were encouraged to answer candidly. Second, the qualitative study repeatedly suggests that students indeed felt incentivized (or, as many said, "forced") to read when they were assigned to the web postings and cold calling group (see Box 2 for some representative quotes.) Third, students protested several times in the semester about having to read and about the amount of reading.
This had never happened in the four previous iterations of Levy teaching the course.
Given that students assigned to the WP&CC group read more before class, one may expect that they both would participate more in class and do better in the problem sets and exam. This is not confirmed by the data.
Although they seem to participate more often in class (0.66 times more), the difference is not statistically significant. Their problem set grades actually decreased by 0.18 points (about 0.43 standard deviations of the control group mean), and while this effect is only significant at the 10% level, it is suggestive that perhaps reading more led students to devote less effort to their problem sets. The qualitative study provides some support for this "crowding out" effect. (See Box #3 for some representative quotes.)

Box #2 -Representative quotes suggesting students read more
 "I only read when I was in the "web posting + warm calling group -sorry!"  "I would not read if I was not forced to through web postings"  "I would not have done readings except for warm calling and postings  "Warm calling and web posting forced me to read"  "Given time constraints, I would probably not have read anything without the warm call motivation. Unfortunately this is part of life @ the Kennedy School"  "Because the web posting was mandatory, they forced me to read."  "I did not do the reading from the textbook when it was not required"  "If no web postings, sometimes you intend to do the readings and end up not doing them"  "Without required posting, I probably would not have read"  "I was "gently" forced to read the material"  "Frankly my reading seriously dropped off in the second half of the semester when I wasn't required to post on the web. Incentives, incentives, incentives"  "The threat of being called in class motivated me to read" Source: Qualitative survey done at the end of the semester Page 13 of 22 Finally, the treatment also did not translate into an increase in learning, at least as measured by the midterm exam score. The effect on test score was small (0.03 standard deviations) and statistically insignificant at conventional levels. There are several possible substantive explanations for this result. The first, perhaps naturally favored by some economists, is that students are skilled in optimizing their use of time. As such, the WP&CC treatment may have led students to do more of something (in this case reading) that was less beneficial per time spent than some other things that got crowded out (in this case, studying or working on problem sets). Thus, an increase in reading, and even in time spent on the course overall, may not translate to better academic performance. A second explanation is that reading did not add significant value to student understanding, perhaps because the reading was not complementary enough to the class or because the reading was not appropriate for the type of students in the course. Several students voiced this general opinion; a representative quote from one: "Readings did not significantly enhance learning beyond lectures." A third possible explanation is that test scores are hard to affect in general, a finding supported by the research literature (i.e. Prince, 2006), and that these interventions were simply not strong enough (perhaps because of their short duration or because of the way they were implemented) to generate a significant impact. A fourth explanation is that since not all questions in the midterm exam drew directly from the materials in the readings, the midterm exam score was perhaps not measuring the learning that students were supposed to attain from the readings. But we conducted some analyses where the dependent variable was the aggregate score on the questions that were unequivocally related to the readings in the textbook 6 , and again found no effect of the treatments on this aggregate score.

7
There are also some methodological alternative explanations to the lack of effect on midterm test scores. The first one is that, given the small sample sizes, the study did not have enough statistical power to detect reasonably 6 To decide which questions fell in this category, we asked Suzanne Cooper, a veteran instructor of the course, to make such determination. This represented an informed judgment call from someone not involved in this study. 7 The point estimate of the coefficient on the treatment variable was -0.08 in Experiment #1 and 0.11 in Experiment #2, and neither of these was statistically significant at conventional levels.

Box #3 -Representative quotes suggesting reading crowded out other course activities
 "Postings and reading did not enhance the in-class learning, rather they took time away from problem sets"  "If you do the readings hastily (because there is so much to do for this course), it does not make much of a difference"  "While the warm calling did nudge me to be more motivated to do the readings, the intense workload of the course and mandatory biweekly postings completely burnt me out and crushed my motivation to read by the end of the course"  "I did not like the web postings because they distracted from focusing on studying the material. " Source: Qualitative survey done at the end of the semester sized effects. While we cannot discard this explanation, we do not think it is very compelling. For one thing, the statistical power of the design was greatly improved by stratifying the random assignment using the grade on the first course of the sequence. Additionally, the point estimate was 0.03 standard deviations, and even with our relatively large standard error (0.17), we should have been able to detect sizable effects in the order of a third of a standard deviation or more. The second methodological explanation is that the treatments may have had spillovers. To the extent that cold calling and web posting led to half of the students being better prepared for class, perhaps the discussion in the classroom was enriched in a way that benefitted everyone in the course (including the half of the students who were in the control group). If this were indeed the case, our estimates here would understate the effect of WP&CC on learning. We cannot discard this explanation but would argue that the spillovers, if they existed, could also have affected negatively the control groups. For example, if the discussion occurred at a higher level (or faster pace), the control group students perhaps could have been more likely to be lost or confused, and if this were the case our estimates here would be overstating the effects of WP&CC on test scores. In sum, while we cannot discard these two methodological issues as partly explaining our results, we think it is unlikely that they are fully responsible for them.

Experiment #2 -Web Postings vs. Cold Calling
When tested against each other, neither of the two methods (Web Posting and Cold Calling) was found to be more effective in terms of improving any of the key outcomes (Table 3b). The differences in average outcomes between the web posting and cold calling groups was not statistically significant for time spent reading, time spent in the course, class participation, problem set grades and midterm grade. The difference in the midterm grade was 0.24 standard deviations in favor of the web posting group, which is not trivial, but it was not statistically significant. Furthermore, given no statistically significant effects on the other four outcomes, as well as the lack of impacts on midterm score in Experiment #1, it seems hard to argue that the web postings have an effect over cold calling with regards to midterm scores.

Student and HKS Faculty Perceptions of WP & CC Effectiveness
The reader could argue that perhaps the results reported above were not all that surprising. We argue here that they were. First, Levy's priors were that the combination of web posting and cold calling in Experiment #1 would have led to higher scores on the midterm exam. After all, he had been using these techniques for two years because he thought they led to increased learning. Second, as described below, samples of faculty and students by and large did not predict accurately the observed pattern of results.
We conducted two surveys to examine how predictions of the effectiveness of the treatments related to the study's actual findings. The first survey group was composed of all the students taking part of the study, while the second included approximately 40 Kennedy School Faculty members attending a research seminar in which this study was presented. In the case of the faculty members, they were asked to make the predictions after they had heard the description of the interventions but before they had heard the results.
In terms of Experiment #1, the vast majority of both groups correctly predicted the positive effects on reading time (92% of students and 74% of faculty). But only a minority of each group (26% in both) correctly predicted the lack of effects on learning (Table 4a). Students and faculty were not much better predicting the results of experiment #2. The majority of them thought that web posting would increase reading time relative to cold calling (Table 4b), which was not supported by the evidence. Furthermore, only 15% of faculty members and 18% of students predicted that web posting and cold calling would be equivalent in terms of their effect on learning.

Conclusions
Many instructors struggle with how to get their students to prepare better for class. This study reports on the effects of two frequently used techniques (web postings and cold calling) on student outcomes in a statistics course that MPP students take in their second semester at the Harvard Kennedy School. When used together these two techniques seem to increase the amount of time that students devote to reading before class by about an hour. This effect is both statistically and practically significant. However this increase in reading did not translate into increased learning (measured by average test scores on the midterm exam). Neither of the two techniques seems to be better than the other one at increasing reading time, test scores and other student outcomes.
Possible substantive explanations for why web postings and cold calling affect reading time but not test scores include: the reallocation of time induced by the interventions was not optimal for increasing learning, the readings themselves did not add much value to other activities in the course, and the treatments were not intense enough to generate impacts on a variable as hard to affect as test scores.
We do not conclude that web postings and cold calling cannot helpful to increase learning in general. Instead, we conclude that they were not helpful in this context as implemented here. Perhaps more importantly, we conclude that instructors using techniques they think are leading to increased learning would perhaps do well to assess whether this is actually true. Both authors, as well as the vast majority of students and other faculty members surveyed, had intuitions that ran counter to the main results of the study, namely that these interventions would have some positive impact on learning. If anything, we hope this study contributes to the notion that gathering evidence about our teaching has the potential of helping us become better in our teaching.
Page 17