Leveraging a Course Quality Checklist to Improve Online Courses

Designing high quality, interactive online courses in a technologically rich environment can be a daunting task even for experienced faculty. This process becomes more difficult when faculty are teaching multiple classes while juggling service responsibilities and research/creative scholarships. In order to help faculty focus on key aspects of online teaching and course design, we developed a checklist with links to institutional resources which help faculty meet several best practices for online teaching. Use of checklists and rubrics to meet quality assurance standards is common (e.g., OLC OSCQR Course Design Review Scorecard, 2018; Quality Matters Higher Education Rubric, 2018), but they vary significantly with the time required to review a course (Baldwin, Ching, & Hsu, 2018). Our goal was to create a checklist that helps faculty design basic elements of the course and expedite the self-review process. Given the current 2019 coronavirus disease (COVID-19) situation, when instructors were suddenly asked to teach remotely, this Course Quality Checklist helped faculty selfreview their existing or new online course via multiple lenses, such as course orientation, policies, organization, and alignment, as well as Universal Design for Learning and interaction. Faculty may use this checklist to create a clear and consistent structure within their course. The checklist also links to several online, just-in-time resources (e.g., course templates, design and pedagogy training, and standards for interaction and accessibility). This will ensure they meet essential standards, save time, reduce cognitive load, and meet specific compliance requirements.


Introduction
In March 2020, the World Health Organization assessed that the 2019 coronavirus disease (COVID-19) is a pandemic affecting more than 114 countries (Ghebreyesus, 2020). This triggered closure of on-campus instruction in affected universities across the globe, and faculty had to teach remotely. Institutions that had a long-standing history of online teaching were able to transition quickly whereas universities who did not have the culture or the resources struggled. This rapid and mass shift to online instructional delivery led to creation of several subpar online classes. All NC-SARA affiliated universities are bound by a reciprocity agreement which ensures that all institutions will meet reasonable quality and compliance standards (NC-SARA, 2020). There are various ways in which universities can meet these quality standards, but most are human resource intensive and timeconsuming. To maintain quality in online course design, one of the effective approaches is to use checklists. This has been a common practice in academia (Heidke, 2015;Rochester Institute Technology, 2020;McGahan, Jackson, & Premer, 2015;Baldwin & Ching, 2019). As the number of online courses increased exponentially during the pandemic, the checklist quickly became a mechanism to ensure that faculty develop online courses that meet the quality standards set by the university. The IU Online Course Quality Checklist assists in development and maintenance of online courses by helping solve challenges associated with online course development. The checklist helps the university scale up faculty training despite limited human resources to train faculty. Developing an effective online course is a complex and time-intensive activity that needs intentional focus. The checklist serves as a guide for faculty as they design their online courses efficiently in response to the urgent need to pivot to online and remote instruction. It is also a management plan that tracks the various aspects of course development.
One of the key barriers to developing a high-quality online course is lack of time since faculty have multiple demands on their time, such as service responsibilities, research and scholarship activities, teaching multiple courses simultaneously, grading, dealing with student issues, and participating in other teaching-related activities. Lack of training on how to develop an online class and absence of clear guidelines are additional barriers to developing a high-quality online class. Absence of effective models for building online classes, lack of organizational infrastructure-both financial and human resource-and a lack of best practices in a plug-and-play format are some of the key reasons why faculty find it a challenge to develop high-quality online classes, especially in situations where a quick turnover is expected. Absence of such resources leads to a patchy implementation of strategy. Also, many faculty can only get to their online classes later in the day when institutional support staff may not be available. This often leads to missed goals, which leads to frustration, especially if instructors are working with an instructional design team. Several institutions develop communities of practice where instructors can learn from their peers (U.S. Department of Education, 2014). In scenarios like the COVID-19 pandemic when physical presence was not possible and such a community of practice could not exist, the checklist may serve as a point of convergence for best practices and lessons learned from several experts across disciplines.
The checklist gives faculty ownership of their course design as it intentionally guides them to meet the criteria as outlined in the checklist. Content experts often have little formal training regarding online pedagogy, which is necessary for online instructors (Salmon, 2014). Most learn on the job how to teach online from peers, leading to inconsistent transmission of best practices. The checklist is one approach to ensure that all online instructors, new or experienced, receive timely and complete information in an applied format.

Advantages of a Course Quality Checklist
One of the unique features of the checklist is that, not only does it highlight the best practices, but it also links to existing university resources. This integration of several optional resources (e.g., IU Canvas Studio, IU QM Syllabus Template, Teaching for Student Success -Canvas Course for Faculty, Technology Tool Finder, etc.) helps faculty by providing just-in-time resources and training. This checklist saves time and is responsive to the needs of faculty new to teaching online by providing templates and models (e.g., modules, home pages, and syllabi designed for online learning) and allowing faculty to quickly and efficiently integrate best practices into their courses. Faculty can continue evaluating their course at their own pace and are not dependent on another peer, making it an asynchronous course development tool. Due to the self-paced nature of the checklist, faculty can continuously reflect on their course design, prevent avoidable errors, and make changes at their convenience until all criteria are met (Moon, 2020).
The combination of the checklist and its embedded resources simplify the process for faculty who are new to designing online classes. The checklist focuses on course design and not on course content, meaning it can be applied to courses from any discipline, modality (100% online, hybrid, and face-to-face classes that leverage the LMS), and level (undergraduate, graduate, and post-graduate). Various departments within a university can customize the checklist to incorporate internal university/department-specific resources based on their university standards, thus empowering their faculty to meet criteria on their own. In its current form, the checklist and the embedded resources (e.g., IU QM Syllabus Template) help meet several external standards as established by agencies such as Quality Matters, OLC, or accrediting bodies. This saves time and effort for faculty and allows them to focus on course content rather than design aspects of the class. This helps reduce the overall cognitive load on faculty, and they can use the checklist to track their progress as they continue to build their course. It also provides the university with a process to evaluate an online course to ensure that it has all the course design elements while streamlining the development and management of online courses.

IU Online Course Quality Checklist
The IU Online Course Quality Checklist was designed around the following four online course design criteria:

Course Orientation & Policies
Providing a welcoming and clear learning environment is critical to student success in online and hybrid courses. This section of the IU Online Course Quality Checklist outlines a few simple course design decisions an instructor can make that helps students better understand how the course is designed, what to do first, and where to find all the course or institutional policies that contain important information in regard to how the course will be delivered. Having students and the instructor introduce themselves at the beginning of the course can also contribute to the creation of a community of inquiry in an online course. At the very least, it helps provide a human connection to others in the course.

Organization
An online course should have a consistent organizational structure to help promote student success in the course. Whether the instructor organizes the course by week or unit, the key is consistency. When the organization changes from week to week, students can become disoriented and become focused on trying to find the instructional materials or assignments for the week, as opposed to being focused on their learning. Including the estimated amount of time students will spend learning and completing assignments each week will help students understand how best to manage their time before beginning a weekly module.

Alignment
The concept of alignment is critical to a well-designed online or hybrid course and its contribution to student learning. Quality Matters defines alignment as, "Critical course components... reinforce one another to ensure that learners achieve the desired learning outcomes. When aligned, assessments, instructional materials, learning activities, and course technologies are directly tied to and support the learning objectives" (2018). The concept of alignment has significant ties to the Backward Design instructional design theory promoted by Wiggins and McTighe (2008), in which the learning outcomes are the foundation of the course design.
First, the learning outcomes should be measurable and specific. This means the learning outcomes should use an active verb that contains a behavior that is observable by the instructor. For instance, consider the learning outcome, 'Students will explain the concept of photosynthesis.' This is measurable because an instructor would be able to observe a student explaining the concept of photosynthesis, but it would be difficult to observe a student's understanding of photosynthesis because the verb 'understand' has various meanings. Therefore, to ensure students are meeting the stated learning outcomes of the course, assessments must be designed so that they directly measure the learning outcomes and capture the observable behavior outlined in the outcomes.
Next, we would want to ensure the instructional materials provided in the course will help students meet the learning outcomes. After we have curated or created the necessary instructional materials, we will want to design learning activities that provide students an opportunity to actively engage with their learning and practice new skills in a low-stakes environment, coupled with formative feedback from the instructor. This formative feedback helps students understand what they are doing well and in which areas there is room for improvement before completing the larger, summative assessments in the course (e.g., term paper, group project, etc.). Finally, the course technology should help rather than hinder student achievement of the learning outcomes.

Universal Design for Learning & Interaction
Ensuring the course is accessible to diverse learners from the beginning of the term is important for student success in the course. Universal Design for Learning promotes the idea that when a course is designed for those in the margin, it benefits everyone. This can be achieved by several ways in the course design. One way is to provide a variety of instructional materials in the course so that information is communicated in a variety of ways to appeal to students' preferred learning methods.
Another way is to ensure that the pages and materials in the course are accessible to those using technologies like screen readers. Even captioning videos helps not only those students who may have a hearing impairment, but also those students who are in environments where listening to the audio is not feasible (e.g., a library, bus, dorm with other roommates, etc.). Providing captioned videos also helps students whose first language is not the one in which the course is offered better learn that language.
Designing purposeful interaction that encourages active learning also promotes student success in online and hybrid courses. Activities like case studies and group projects can help students learn from one another, actively engage with the course content through application, and promote several soft skills students will need to be successful in the current economy. Regular and substantive interaction with students initiated by the instructor can also help ensure student success in the course, as well as meet federal regulations in the classification of distance learning courses.

Faculty Interaction
Determining the full extent of how many faculty have accessed and used the IU Online Course Quality Checklist is somewhat difficult to ascertain, but we do have some concrete data. Currently, the checklist is shared with the university community as a webpage to ensure that it is fully accessible to all. According to our analytics for the webpage, the checklist has 172 unique page views. This checklist has also been shared at various virtual conferences through Google Drive (which does not provide access/downloadable data). During one such presentation at the QM ConnectLX conference in October 2020, 51 attendees were present and had access to the checklist in Google Drive.

New Needs Fulfilled
While our initial purpose of developing this checklist was to cut down the amount of time faculty spend designing online courses, other uses for the checklist have emerged over time. At IUPUI, the checklist has been successfully used to reduce the time to peer review an online course that is preparing to undergo an official Quality Matters review. Typically, the internal review process takes about four to five weeks, but when the checklist was used, the review process was shortened to one week. The review was performed by a member of the Center for Teaching and Learning, and to ensure the feedback provided through the checklist was valuable and actionable, space was provided under each item to provide more detailed feedback or suggestions for improvement.

Conclusion
The IU Online Course Quality Checklist was developed by the Office of Collaborative Academic Programs and reviewed by the eLearning Design and Services office (the instructional design team at Indiana University), as well as the Directors of the various Centers for Teaching and Learning. With this self-review tool, faculty can evaluate whether their online course meets university standards and can be helpful in situations when universities must urgently transition to online modality. By using the checklist, faculty and staff can build upon the collective experience of faculty who have taught online courses before. The checklist is freely available to the public and can be accessed via the TeachingOnline.IU webpage or this Google Drive link. The same checklist is also available in the accompanying Appendix.

Target Audience
The target audience for this checklist are all online instructors, deans and program coordinators at Indiana University. This checklist also links to the following useful resources that are freely available to IU faculty: 1. IU Canvas Studio 2. IU QM Syllabus Template 3. Teaching for Student Success -An Evidence-Based Approach Canvas Course 4. Technology Tool Finder 5. "How do I Make my Class Accessible?" document 6. IU Interaction Standard 7. Creating an Accessible Syllabus using Microsoft Word Instructors are not limited to the resources above. We encourage faculty to work with their Center for Teaching and Learning for templates and other helpful resources, as well as one-on-one support. If you choose to use the QM Syllabus Template document for your course syllabus, you will meet 10 QM-standards out of the 42 standards.
If you have questions regarding this document or the QM-certification process, please consider reaching out to your Center for Teaching and Learning for additional information.

IU Online Course Quality Checklist Course Orientation & Policies
The course provides clear instructions on how to get started in the course, locating key course components like the syllabus and course schedule. You can find course homepage templates in Canvas Studio. Instructor and students introduce themselves at the beginning of the course. Syllabus posted in Canvas one week prior to beginning of the term with the following information. You are welcome to use the QM Syllabus Template as a resource.
Course title and number. Semester, year, and course section. Credit hours. Name(s) of instructor(s) and contact information for instructor (e.g. email address, phone, Zoom room, office location, etc.). Office hours -indicate when and how you will grade/provide feedback, respond to student questions, meet with students online, etc. Course description from catalog. Demonstrate alignment between course & module learning outcomes and course assessments. A table is one method of demonstrating alignment.
Link or reference with specific department, program, general education, or program outcomes, if applicable.
Required and optional course materials (e.g. textbooks, webcam, microphone, software, etc.). Grading policy (breakdown of all assignments with point/percentage value for each, grading scale, expected instructor response time to grading assignments and providing feedback, and late work/makeup exam policy). Student participation and etiquette expectations (e.g. netiquette policy). Link to IT Helpdesk for student technology support. Minimum technology requirements for the course.
Links to accessibility statements and privacy policies for course technologies. Links to campus-specific policies, including academic integrity, grade appeal, etc. Links to academic support services, such as a writing center or math tutoring. Link or statement regarding students with disabilities and acquiring accommodations.

Organization
Explicit and consistent organization and pace of the course (e.g., use of modules with a weekly landing page and assignments located in modules). You can find examples of modules in Canvas Studio.
Include estimated time students should spend on course material each week.

Alignment
Course learning outcomes are measurable, specific, stated from the student's perspective, and align with the course description.

Universal Design for Learning & Interaction
Course design reflects Universal Design for Learning principles (e.g. heading styles used to create structure in pages and documents, accessible tables, alt-text or transcripts for images, closed captions or transcripts for videos, scripts for narrated PowerPoints). You can learn how to incorporate accessible course design in the How do I Make my Class Accessible? document. Also, see the Course Design module within the Teaching for Student Success course to learn more about Universal Design for Learning. A variety of instructional materials and technology tools used in the course (e.g. textbook, scholarly articles, videos, podcasts, simulations/games, etc.). Interaction statement that describes how and when the instructor will regularly interact with students in the course, including response time to emails and other communications. You can learn more about IU interaction requirements in the IU Interaction Standard document. Description of faculty-initiated interaction that will be substantive (relating to the course content) and regular (weekly). You can learn more about IU interaction requirements in the IU Interaction Standard document. Student-to-student interactions support active learning (e.g. discussions, group work, etc.). You can learn more about creating engaging interaction in the Effective Student-to-Student Interaction module within the Teaching for Student Success course (coming soon).