Introducing a Pre-Determined Feedback Pool for Evaluating Students’ Online Discussion Participation

As self-regulation for successful online learning is a relatively new focus in online education, more attention is directed toward initiatives to help students become more cognizant of their learning efforts in online learning environments. Grounded in the idea that self-regulation is feedback dependent, we argue for the necessity of instructor feedback on student performance in online classes. The specific focus is instructor feedback on students’ online discussion contributions. To assist instructors in managing their feedback practices in large online classes with numerous discussion entries, we present a feedback pool that includes a range of predetermined comments instructors can choose from while assessing their students’ discussion participation.

self-regulation?Although not specific to online discussions, in their description of motives to selfregulate learning, Zimmerman and Clearly (2009) made the argument that self-regulation is guided by feedback, and they identified three feedback sources that trigger self-regulation: "social (e.g., praise or guidance from a teacher, a peer, or a parent), environmental (e.g., task, microenvironment, or computer outcomes), and personal (e.g., awareness of covert, physiological, or behavioral outcomes)" (p.247).We concur with Johnson and Johnson (1993) that "while the optimal situation may be to receive feedback concurrently [from all three sources], the most powerful source of feedback is from other people," (p.136) and in the rest of this paper we turn our attention to a specific form of social feedback -feedback from the instructor.
Online teaching and learning literature recognizes feedback from the instructor as an important part of the learning process: "Feedback from the instructor is an important ingredient to the process because it lets students know when their thinking is on track and when it needs revised or redirected" (Lear, 2010(Lear, , p. 1424)).Here, we take this argument a step further and connect it to self-regulation and online discussions by stating that feedback from the instructor serves as a powerful means of directing students' attention toward the desired participatory behaviors in online discussions.
It is no surprise that not all students display the same participatory behaviors in online discussions.While some students mindfully engage with discussions, paying close attention to the details of their performance and being self-observant (aspects of self-regulation), others need to be nudged into doing so (Perera-Diltz & Monaghan, 2014).We argue that individualized instructor feedback during discussions can benefit both groups of students: Feedback indicating deviation from desired expectations may help the latter group seek corrective actions; and feedback indicating successful performance may encourage the former group to continue their efforts to plan and reflectively monitor their discussion participation and behavior.
As Tan, Ullrich, Scheuer, Melis and Shen (2011) pointed out, some online classes have several hundred students.In such large classes, providing specific feedback that can bring about mindfulness in students and influence their self-regulated engagement with discussions can be a challenge for instructors.The contribution of this paper lies in its attempt to address this challenge by providing instructors with a feedback pool from which they can pick and choose appropriate comments and/or feedback applicable to various student behaviors when assessing discussion participation in large online classes.
The feedback pool we report here was developed through qualitative coding and analysis of discussion transcripts generated by students in fully online, graduate-level counselor education courses.The pool contains 47 comments subsumed under two feedback categories: growthoriented feedback and feedback for maintenance of desired behavior.In the sections below, we first describe the methods used to generate the feedback pool followed by a discussion on its significance, limitations along with future directions.

Methods
When we set out to develop the feedback pool, our interest was to access a large volume of student discussion data that could lead to a comprehensive analysis.Our context was an online counselor educator program at a public university in Southeast Texas.We began by identifying courses within this program that could yield rich student discussion data.Our search led us to two courses: Professionalism, Ethics, and Law; and Cross Cultural Counseling.Both courses were 5-weeks long and relied heavily on weekly discussions (each week had a discussion component).After obtaining Institutional Review Board approval, we accessed the student discussions generated in three sections of the Professionalism, Ethics, and Law course and three sections of the Cross Cultural Counseling course that were archived (offered in the 2012-2013 academic year).Each section (a total of six) consisted of approximately 20-25 students.The three sections of the Professionalism, Ethics, and Law course was taught by three different instructors but the learning outcomes, content, assignments, textbooks, and discussion activities were the same in all of them and they all followed the same syllabus.Their look and feel in Blackboard Learn (the learning management system that was used to deliver instruction) were also the same.This was the case for the three sections of the Cross Counseling course as well; they were taught by different instructors but had the same syllabus, same look and feel, same content, assignments, textbooks, and discussion activities.
In all sections, we focused on the discussions generated by students at the midpoint (Week 3) of the 5-week term.Our reasoning was that by this point in the term the students got accustomed to the structure of these courses and were able to sustain discussion without much prompting from the instructors.
The discussions in Week 3 yielded approximately 840 total pages of transcripts, roughly 130 to 140 pages per section.In the three sections of the Professionalism, Ethics, and Law course, the discussion for Week 3 was based on a scenario that required students to: (a) demonstrate decision-making using the ethical and legal guidelines they studied that week; and (b) describe their feelings during this decision making process.In the Cross Cultural Counseling course, the discussion for Week 3 required students to: (a) provide a synopsis of a movie that has a multicultural element; (b) identify positive and negative portrayals of different populations in the movie; and (c) discuss the degree to which the movie matched the information students studied that week about counseling American Indian and Asian American clients.(See Appendix 1 for full description of discussion prompts).

Method of Analysis
In the 840 total pages of discussion transcripts that we gathered from Week 3, we grouped the posts or messages from individual students together and labeled them as blocks.These blocks became our unit of analysis.We used inductive coding to analyze the blocks.Rather than analyzing the discussion transcript for each section in its entirety (i.e., 130 to 140 pages), we stopped coding when no new codes emerged due to saturation (Charmaz, 2006).We typically achieved saturation between pages 40 to 50 and these pages typically represented a minimum of 10 blocks (i.e., posts or messages by10 students).When we reached saturation in one section's transcript, we moved on to the next section's transcript.Using this method, we analyzed approximately 300 total pages of data.
We conducted the coding (using NVivo 10 qualitative analysis software) in two phases: Initial and focused coding (Charmaz, 2006).In initial coding, we read and reread the posts in each block to identify salient patterns, following Ryan and Bernard's (2003) useful list of things to look for, such as recurring regularities, similarities and differences across posts, and missing data (posting behaviors that students either intentionally or unintentionally avoided).Focused coding emerged from constant comparison of the initial codes with data.During focused coding, we made decisions to collapse or assign a new category for coded material.Below are two examples illustrating our coding process.
Example One.The first example comes from Sally (a pseudonym), a student from one of the Professionalism, Ethics, and Law sections.The block below contains her response to the scenario-based discussion prompt and the two other posts she made in response to other students.
Post with appropriate feedback.The first area that needed improvement in Sally's posting practices was her underdeveloped writing.In all three responses, Sally made general statements that did not provide any descriptive detail about the ideas she wants to convey.For example, in the first post, when talking about her options, she wrote she would "convince her [the client] to include her mother and gain her mother's support."Sally did not give her readers any specific information or illustrations detailing what exactly she would do to convince the client.Similarly, Sally's last post was underdeveloped.Her statement, "I would be sure to document the whole thing in my counselor notes," did not give readers any indication of how or in what manner she will conduct this documentation.In her posts, Sally also appeared to make unsupported assertions without proper citation or evidence.She based all of her posts on personal opinion without making connections to the ideas/concepts that were presented in the reading assignment.At one point, she made a reference to a HIPPA statement, but rather than citing the statement and exploring its significance to the topic at hand, she moved ahead with personal opinion.Lastly, Sally's posts at times suffered from grammatical and mechanical errors, such as incomplete sentences (e.g., "Prior to sending I would attempt to …") incorrect spelling (e.g., "responsiblity" "dilemna" "confidentialy") and punctuation errors that become distractions for the reader.Such problematic elements in Sally's posts can be targeted and discouraged through instructor feedback so she may consciously improve her discussion responses.Aside from these, there are aspects of Sally's posting practices that are laudable.One is that she showed engagement in the flow of the discussion by posting on different days.Students were required to distribute their posts evenly throughout the discussion period.She also met the minimum requirement for three weekly posts (minimum three substantive posts were required).These behaviors conducive to learning can be encouraged through instructor feedback.
Example Two.The second example comes from Joe (also a pseudonym), a student in one of the Cross Cultural Counseling sections.The block below contains his response to the discussion prompt (based on a movie) and the two other posts he made in response to other students.
Post # 1: Date: December 5, 2012 Until I read the text for this week, I did not realize that the term "model minority" had been coined to refer to high achieving Asian students.For years, I was fascinated by my Asian students who seemed to be more focused, more responsible, more accountable, and more respectful than their peers of other backgrounds and cultures.I can remember writing a pretty lengthy paper on my Asian students and their culture while working on my certification through UTA.I wanted to know then what aspects of their culture lead them to set such high social, behavioral, and academic standards for themselves as a race of people.Baruth & Manning (2012) state that the Asian culture emphasizes "diligence, harmony, taking responsibility, respect for authority, emphasis on education, respect for elders, and family loyalty."Although these attributes are important and generally instilled in Asian children, I have learned over the years that my thinking of Asian students as model minorities is stereotypical.Whereas I used to believe that every Asian student that I met would be high achieving and have a strong work ethic, I have come to find after 13 years of teaching that that simply is not true, and it is an "unrealistic expectation" and not fair to my students to make that assumption (Baruth & Manning, 2012) Joe provided analytical and reflective responses in his first post.For example, in the first paragraph, he compared new knowledge (gained from the course readings) to past experiences (gained from personal experiences) and this comparison allowed him to approach the discussion content with considerable thought and reflection.In the second half of his first post, Joe explored and analyzed the movie in relation to the knowledge gained from the assigned readings.In his second and third posts, Joe took an interactive approach.He began with an affirmation of what others said and then developed the ideas with personal opinions and interpretations.Next, he took some of the ideas he read in the course readings and extrapolated them to what works in practice.His ability to move from personal exploration to idea extrapolation exhibits thoughtful and logical sequencing.Joe also showed engagement by posting on three separate days as the discussion unfolded rather than making all his contributions on the same day.Overall, Joe displayed posting behaviors that are desirable and should be reinforced with appropriate feedback.

A Tentative Model of a Feedback Pool for Evaluating Students' Online Discussion Participation
Our qualitative analysis (which we illustrated with examples above) resulted in a tentative model of a feedback pool (See Table 1) that online instructors can use when evaluating students' discussion participation.The pool includes two types of feedback: growth-oriented feedback (i.e., feedback-indicating deviation from desired norms or expectations) and feedback for maintenance of desired behavior (i.e., feedback-indicating alignment with desired norms and expectations).

Significance
In large online classes, providing students with specific and individualized feedback that promote their self-regulated engagement with discussions can be a challenge for instructors.In this paper, we propose a feedback pool as a potential solution to this problem.We believe that the availability of a multitude of predesigned comments to choose and personalize from can transform online instructors' feedback giving practices, including the content of the feedback as well as its timing, efficiency, and management.More importantly, receiving feedback not just a grade will potentially influence students' intentional self-regulation, leading them to monitor their discussion performance and strategically plan for improvements.
Online instructors interested in using this feedback pool can utilize it in the way that best serves their needs.They can personalize the feedback comments or change them as appropriate for individual students.When using the feedback pool, we recommend that instructors gauge how much feedback to provide and which categories to focus on.For both instructors and students, selective and focused feedback (i.e., feedback focusing on two or three categories rather than all or most categories all at once) will be more manageable.A limitation of the feedback pool presented here is that it was developed with discussion data obtained from counselor education courses.However, since the comments presented in the pool address various forms of discussion participation behaviors that are not subject-specific, they can be used in online courses in disciplines other than counseling.Perhaps one can argue that the comment pool is more appropriate for graduate level courses, especially those in social sciences.One way to see whether this is in fact the case is to ask this question: Is the comment pool going to be used for discussions requiring students to show understanding and synthesis of ideas and to collectively build on each other's ideas?If the answer is yes, then the pool is very likely to be applicable.

Limitations
Another limitation is about the caveats to consider when using feedback from a pool of pre-determined comments.First and most important, when students repeatedly receive the same feedback (in which the same words are used) or receive the same feedback as others, they may start perceiving such feedback as canned, rather than authentic.As such, while pre-constructed feedback pools are helpful for instructors with large class sizes, they must be carefully utilized so as not to give students the impression that their performance is not monitored and evaluated with scrutiny.Secondly, no pre-determined feedback pool can be exhaustive.There may be instances where the comments in the pool we provide will not suffice.Thirdly, when using comments from a feedback pool, instructors need to make sure that the comment they select from the pool actually reflects the appropriate or intended message that needs to be communicated to a student (e.g., instructor indicates a change should be made in a student's posting behavior when in actuality that change is not needed).

Future Directions
In a recent Faculty Focus article on tips for effective and efficient grading, Smith and Palenque (2015) suggested, "Keep a bank of comments about frequent errors students make and organize them in groups for easy access."The feedback pool presented in this paper is an actual realization and implementation of this recommendation.To improve on the work that has begun with the generation of this feedback pool, there is a need to develop a feedback plug-in application for any learning management system to allow instructors to drag and drop appropriate comments from the pool and insert them into their evaluation of individual students' performance on discussion.

Connecting experience to learning
Use personal experience or anecdotes judiciously.If appropriate (and only for illustration of something), relate the topic to your personal experience and be sure to make the link between the experience and the course material.
Your judicious use of personal experience (or anecdotes) enriched your writing and allowed you to make meaningful connections between the course material and the real world.

Making assertions
You have stated information as factual or made sweeping generalizations without proper support or evidence.Your writing would be enhanced if you supplied reasons for your opinions or explained how you came to your ideas(reached your conclusions).
In your post, you have supported your own points/opinions with relevant facts and evidence.This allows you to justify your reasoning and enhances the accuracy of what you have written.

Relevance
Your writing strays off topic into areas that do not relate to the topic of discussion.Please only include material that relates to the topic at hand.
You write honing in on the topic.This helps the reader understand your insight related to the topic.

Analyzing
Your writing would be strengthened if you engaged in some analysis by identifying or examining connections, patterns, differences or similarities between the text and your experiences OR between/among the ideas presented.
By making connections between the text and your experiences and/or extending the ideas by identifying the similarities and differences in them, you are displaying high-level thinking which elevates the discourse.

Illustrating
Your posts would be strengthened if you utilized examples that illustrate the points you wish to make rather than simply listing information or generalizations about a topic.
Your exploration of the course content through the use of examples in your post enhances your writing and facilitates understanding.
Probing Your posts would be strengthened with probing questions that move the discussion forward.
The thought-provoking questions you asked in your post(s) enhanced the discussion and engaged your peers.

Engagement Responsive interaction
Your discussion contributions would be strengthened if you engaged more actively with your peers by building on or inviting their comments.
You are actively engaged with your peers by building on or inviting their comments.
Posting correctly in the body of your post(s) and in your reference list.

Formatting 3
The citations within the body of your post(s) generally occur outside of sentence punctuation.Please make sure to place citations at the end of a sentence before a period.When you place a citation within a sentence, make sure that it appears before internal punctuation such as a comma or semicolon.
Formatting 4 Please remember that every source in the reference list must be referred to at least once within the body of your post.If no citation appears in your writing, a resource cannot be included in the references.
Peer Responses: Multiple posts are made offering substantial, well written contributions, opinions, observations, questions, experiences, critiques, suggestions, etc. Citations must be included.Comment on ethical codes or legal precedents that might relate to your peers' personal experiences.Peer responses must occur on at least two separate days.
Remember: In order to meet minimum requirements, you must respond with your initial post by day 4 (Thursday) and respond to at least 2 peers with substantive responses and support these responses with references.Posts should be evenly distributed throughout the discussion period, reflecting participation from start to finish.All posts should be appropriate, using proper language and mechanics.
Week 3 Discussion Prompt in Cross Cultural Counseling Course For this week's discussion forum, respond to the following questions: This week you read about counseling American Indian and Asian American clients.You were exposed to the various issues that impact the counseling process for different populations.As part of preparing for this week's discussion, read your course materials and then watch one of the movies listed in your course resources for this week (file name is Multicultural Movies_).You can access most of the movies via Amazon Instant Video, Netflix, or any other video source.You can also purchase the movies via Amazon and other vendors.
For this week's discussion forum, respond to the following prompts/questions: 1. Provide a synopsis of the film you watched and the top three multicultural issues that stood out for you.
Perera-Diltz, Davis, Smith, and Sheperis Journal of Teaching and Learning with Technology, Vol. 7, No. 1, Spring 2018 jotlt.indiana.edu95 2. Provide specific examples in which the film influences (positively or negatively) perceptions of multicultural issues.Be sure to identify any potential for the movie to promote bias.3.Discuss the degree to which the film you watched matches the information you have learned in this courseUse examples to support your ideas.Be sure to cite your points using the learning resources.You may also use other scholarly literature to support your discussion.Citations and references must be in APA format.See the grading rubric for more information.Peer Response: Respond to a MINIMUM of 2 peers with substantive responses and support these responses with references.Peer responses must occur on two separate days.See the grading rubric for more information.Before responding to your colleagues, watch one of the films on Asian Americans or American Indians.Compare and Contrast the attitudes, beliefs, understandings, and acculturative experiences inherent in these films to those in the main post.Also contribute to the discussion by building on examples or providing your own in relation to their main discussion post.Posting for the discussion board are due by Day 4, and responses are due by Day 6.
low performing and underachieving, it is not right to think that all Asian students are academically gifted.In The Joy Luck Club, first generation Asian-American children struggle with their mothers over traditions, beliefs, and expectations.There are several ongoing themes throughout the movie: love, honor, respect, obedience, shame, forgiveness, perseverance, choices, and sacrifice.All of the women in the movie have gone through tremendous trials that have affected their lives in a multitude of ways, and they struggle to cope and make sense of their experiences.In the movie, many aspects from the text regarding the Asian culture are recognizable.For instance,Baruth & Manning (2012)note that many Asian children become acculturated and adapt "to the norms and expectations of the dominant group."This is evident in the marriages of the daughters to white men despite the reluctance and concern on both sides.Acculturation is also noted in choosing Americanized names such as She feels that it is inappropriate for Calpurnia, a black woman, to be so familiar with the family, so she moves in to set things straight.Her intolerant, overbearing, racist mentality prompts her to destroy the close bond that her niece and nephew, Scout and Jem, have formed with the only mother-figure they have really known.For years "African-Americans have fought to overcome cultural dominance and discrimination" in situations such as the one described, and although so much has improved from efforts like the Civil Rights movement, racial prejudice still exists in our society(Baruth & Manning, 2012).Counselors can help by advocating for people from all races and cultural groups in an effort to continue moving forward and help to increase tolerance, understanding, and equality for every citizen.The Color Purple, in my opinion, is one of the best novels of all times, and the movie is a great extension of the author's creation.Oprah Winfrey's character brings tears to my eyes every time I watch it.After standing up for herself against a white woman, she is jailed, separated from her children, and forced to become the woman's servant.Racism ultimately destroys her life as she rots away . I have encountered Asian students with varying academic ability levels, personalities, and behavioral patterns.Just as it is not fair to assume that Perera-Diltz, Davis, Smith, and Sheperis Journal of Teaching and Learning with Technology, Vol. 7, No. 1, Spring 2018 jotlt.indiana.edu92 all blacks are Waverly and Jennifer instead of traditional Chinese names.Rose hsu also states that she is "American" when speaking with Ted's mom, as if rejecting her cultural heritage to embrace her American roots.In addition, I noticed a sense of male dominance in the movie that the text references.Baruth & Manning (2012) state that "children learn early on that the father is the head of the household in the traditional Asian American family."Furthermore, the text indicates that Asian people often do not seek counseling intervention when they are having mental health issues.It indicates that many Asians feel as if "talking about problems to outsiders is considered taboo," simply deny" that mental health issues exist, and do not receive help from family members (Baruth & Manning, 2012).Throughout the course of the movie, communication between the family members is poor, and although the women truly have reasons to seek help, they do not make a decision to do so.Finally, it is evident that high expectations are placed on the children from a very early age.Whether it was a piano recital or chess tournament, the children were expected to practice, perform, and compete at high levels as to not shame the family.This coincides with the text's assertion that "Asian American parents who "have expectations of their children are unwilling to negotiate the terms" (Baruth & Manning, 2012).lawyer and works long hours, the housekeeper, Calpurnia, takes on a maternal role for the children in the household.There is a tremendous amount of reciprocal love in the home; however, some people, namely Atticus' sister Aunt Alexandra, disapproves of the relationship.I enjoyed reading your post and learned a great deal from it.I watched A Time to Kill many, many years ago, but feel inclined to watch it again after reading your synopsis.It is often very difficult for me to read books or watch movies that deal with racism and discrimination.The abuse, injustice, and pain that is visited upon decent, law-abiding citizens simply because others are ignorant and intolerant is hard for me to comprehend.Two other movies came to my mind after reading your To Kill a Mockingbird and The Color Purple.To Kill a Mockingbird is one of our required 9th grade texts in my district, so every year, I get the opportunity to share insight with my students on the destructive nature of racism and discrimination.In this text, a hard working black man, Tom Robinson, is wrongly accused of raping a white woman in the South during the 1930's.And he, too, must rely on a white lawyer to vindicate him so that he can return home to his wife and children.Although things do not end well and he's subsequently found guilty and later killed in jail, his lawyer, an upstanding and honest southern gentleman, risks everything to clear Tom's name and teach his children lessons in integrity, sacrifice, and morality.Journal of Teaching and Learning with Technology, Vol. 7, No. 1, Spring 2018 jotlt.indiana.edu94 Baruth, L. G. & Manning, M. L. T. (2012).Multicultural counseling and psychotherapy.Boston, MA: Pearson.