Evaluation and revision of an introduction to experiential rotations course

: The objectives of this study were to evaluate the perceived student value of topics taught in Butler University’s Introduction to Experiential Rotations (RX500) course, implement course revisions to address any perceived weaknesses, and to reassess the course following implementation of those course revisions. Advanced Pharmacy Practice Experience (APPE) students from Butler University’s 2012 doctorate of pharmacy class were initially surveyed to assess the perceived usefulness and design of RX500. Based on the findings of the initial data, course revisions were developed and implemented for the following semester’s RX500 course. In order to assess for potential changes in the perceived value of the course following implementation, a follow-up survey was sent to students completing the revised course in the fall of 2012. Fifty-eight students completed the initial survey in the fall of 2011, and 34 completed the follow-up survey in the fall of 2012. The majority of students in both groups had completed at least four APPE experiences at the time of survey completion. Both of the study groups rated hands-on activities and visual presentations as their most beneficial learning styles within the classroom setting. Based on the initial survey findings, two topics were removed from the course, two topics were revised within the course, and three topics were added to the course. The follow-up survey results indicated an increased perceived usefulness of the two revised topics; however, the three new topics added to the course had mixed results. Faculty can continue to work on delivery of newer topics being added to the APPE preparation course.

to provide a wide exposure to various patients and disease states, highlighting the impact of pharmacists in many different settings. This final year is often considered a capstone year and an opportunity to put to practice knowledge gained during the previous curricular years. Two noteworthy publications within pharmacy education literature illustrate use of a survey tool called the Perceptions of Preparedness (PREP) survey (Reid, 2002& Scott, 2010 to assess student perceptions of preparedness prior to these clinical APPE. The overarching goal of the PREP survey was to utilize information gained to improve the pharmacy program curriculum prior to APPE. The first instance of the PREP survey was published by Reid, et al. (2002) at the University of Florida. Pharmacy faculty created 41 competencies essential for completion prior to the APPE year. These competencies were used to develop the PREP survey. Student perceptions of confidence in each of these competencies were evaluated using a seven-point Likert-scale. To evaluate the results, investigators divided the competencies into five factors: administrative, communication, psychosocial, research, and technical. The study was conducted for four years from 1997 to 2001. Student responses revealed that students were most confident with the communication, psychosocial, and technical competencies, while students reported being least confident with the administrative and research competencies.
Similarly, Scott (2010) and colleagues at North Dakota State University published results of a modified PREP survey (some competencies that were not relevant to the pharmacy program were removed). Students were asked to rate confidence using a five-point Likert-scale, and responses were assessed using the same five factors. The surveys were administered for four years from 2004 to 2008. Students at North Dakota State University were most confident with the communication and psychosocial competencies, while they expressed lower confidence in the administrative, research, and technical competencies.
Not surprisingly, both studies found that graduating students become more confident as they progressed through the pharmacy program and that student confidence in a given area increased following completion of a course that addressed that specific competency (Reid, 2002;Scott, 2010). Additionally, assessments of students from different graduating classes at the same point in the curriculum (e.g. comparing first year pharmacy students from each year of the study) revealed increasing confidence year by year. According to the authors from both studies, this demonstrated the positive impact of curriculum changes made during the four years of the study.
Outside of the PREP surveys, Ho et al. (2009) published positive student perceptions following a curricular overhaul. The Chinese University of Hong Kong School of Pharmacy restructured and revised their curriculum by planning specific learning outcomes for their graduates. The initial curriculum restructure was developed by the curriculum committee with revisions and implementation done by all faculty members. In addition to course specific student focus group feedback, perceptions were gathered via the graduate student survey. Students were asked to evaluate 13 curricular competencies, using a five-point Likert scale. This survey was administered annually within 6 months after graduation. Data was collected over a four-year timespan, starting with the old curriculum and finishing with two years of new curriculum graduates. One hundred eleven graduates (95%) completed the survey and significant improvement was noted in 10 of 13 professional and generic competencies, including critical thinking and interpersonal skills. No statistically significant improvement was noted in computer literacy, problem solving, or self-managed learning. The authors noted overall success in the implementation of their outcomes based curriculum, utilizing student perceptions as their primary method of assessment. Dy, E.A., & Nisly, S. These studies all demonstrate the use of survey-based studies of students' perceived value as an assessment tool when restructuring a pharmacy curriculum. However, the aforementioned studies outlined changes to an entire curriculum and tracked students throughout four years; the goal of this current study was to capture a snapshot of a single course.
The nearest course specific revision comes from Auburn University Harrison School of Pharmacy (AUHSP) (Stevenson, 2011). As a subset of continuous quality improvement within the curriculum, the APPE were reviewed by an experiential subcommittee, built within the larger curricular committee. Prior to assessment of the APPE, materials were collected, including syllabi, orientation materials, and course evaluations. Additionally, a 22 item survey was sent to fourth year pharmacy students completing rotations during the last academic month of their professional program. The survey questions were derived from concepts outlined in the American College of Clinical Pharmacy experiential education White Paper and Position Statement (Haase, 2008a;Haase, 2008b). While the number of students completing the survey is not reported, the authors report that student responses were positive when asked about exposure to multiple patient populations and disease states. Students also positively reported opportunities for collaboration, self-directed learning, and access to drug information resources. Students did identify areas for improvement, including additional electives, clearly defined objectives, and provision of constructive feedback. In conjunction with other materials gathered and faculty focus sessions, specific areas for improvement in AUHSP's APPE were reported. These areas included: further identification of elective opportunities, increased use of the standardized evaluation form, and consistent use of syllabi templates with incorporation of clear objectives. The authors demonstrate effective use of student feedback in restructuring APPE at AUHSP.
Based on a review of current literature, this study appears to be the first research evaluating the impact of student perceptions on a single didactic course within a college of pharmacy. This study was conducted at Butler University, a private university located on the north side of Indianapolis, Indiana (Butler University, 2012). Housed within Butler University's College of Pharmacy and Health Sciences (COPHS), the Doctor of Pharmacy program is comprised of two, liberal-arts based pre-professional phase years and four professional phase years. COPHS students take part in 10, 4-week APPE during the fourth professional year. Clinical sites for APPE include statewide health care facilities, as well as opportunities for travel to national and international sites.
The Introduction to Experiential Rotations (RX500) is a required course within the COPHS pharmacy curriculum (Butler University, 2012). This one-credit hour course is taken during the spring semester of the third professional year, prior to beginning APPE. RX500 is primarily lecture-based and is taught by various university faculty and guest lecturers. Student assessments and grades are determined by attendance, assignments, and two examinations focused on 200 important medications.
The goals of the RX500 course are as follows (RX500, 2011): 1. To introduce students to, and prepare them for, the concept and practice requirements of the Advanced Pharmacy Practice Experiences (APPE).

To facilitate scheduling of rotations and ensure completion of necessary forms, certifications and tests that need to be completed prior to starting rotations. 3. To make students aware of the guidelines and expectations for [required seminars during the APPE year].
Traditional topics discussed throughout the semester range from practical skills (e.g. PowerPoint presentation tips) to reviews of clinical information (e.g. geriatrics overview) (see Dy, E.A., & Nisly, S.A. Journal of the Scholarship of Teaching and Learning, Vol. 14, No. 3, August 2014. josotl.indiana.edu 37 Appendix 1 for full topic list). Topics covered in this course are those not covered in other areas of the PharmD curriculum. The majority of topics are covered within a one-hour course meeting; however, some topics are covered in only a small portion of the 50-minute class meeting, while others span over the course of two 50-minute class sessions.
At the time of project conception, COPHS faculty and administration were re-assessing the Doctor of Pharmacy curriculum. Although RX500 had been in existence for over ten years, its content and structure had been largely unrevised since its inception. As a result, this course was identified as one that might potentially benefit from revisions. Because the course is largely focused on student preparation for the APPE year, the goal was to understand student needs in terms of APPE preparation and to revise the course accordingly. This study utilized APPE student perceptions of RX500 to change the content and delivery of information within the course.

Objectives
The objectives of this study were: • To assess the perceived student value of the topics within the RX500 course • To implement course revisions to address any perceived course weaknesses • To reassess the course following completion of course revision implementation

Methods
The study was designed to be completed in three phases: an initial survey phase, an implementation phase, and a follow-up survey phase (see Figure 1). These three phases are outlined in detail below. All research and study design was approved by the Butler University Institutional Review Board.

Initial Survey Methods
All COPHS Doctor of Pharmacy students, most recently completing the RX500 course in the spring of 2011 (the Pharmacy Class of 2012), were invited via email to participate in the study. Email invitations were sent in September of 2011 and included a link for survey completion via SurveyMonkey™(Appendix 1). Students were not given an incentive for participating in the study but were asked to participate in order to assist in future course revision. The survey first asked students to rate the spring 2011 RX500 topics as "Not Useful," "Useful, Needs Improvement," or "Useful As Is." The second question assessed student opinions on the need for additional topics. Students were allowed to select from a list of potential topics generated from ACPE Pre-APPE Performance Domains and Abilities and from other pharmacy education literature, or they were allowed to provide open-ended responses in a blank space provided (American Council for Pharmacy Education, 2011). Students were then asked to provide information on their learning style, number and type of rotations completed to date, and previous pharmacy work experience. This information was gathered in order to assess any potential differences between the pre-intervention and post-intervention groups.

Implementation Methods
Initial survey results were compiled and analyzed prior to the spring 2012 course. Topics were identified to remove, revise, or add to the course. All decisions were made in conjunction with RX500 faculty course coordinators.

Follow-Up Survey Methods
Students in the Class of 2013 were given a similar SurveyMonkey™ survey in September 2012 to assess potential changes in perceived value of the course. The survey was modified to account for changes to the spring 2012 RX 500 topic list.

Statistical Analysis
Data were initially summarized using SPSS software, version 19.0 (IBM Corp., Somers, New York). Descriptive statistics were utilized to evaluate the results from the 2011 and 2012 surveys individually. For topics revised in 2012, survey results from initial and follow-up surveys were analyzed using Mann-Whitney statistical analysis. A positive change in perception was viewed as a majority of responses indicating progression through the three categories (from Not Useful to Useful, Needs Improvement, from Useful, Needs Improvement to Useful As Is, etc.), and vice versa.

Study Population
A total of 58 out of 116 (50%) students from the class of 2012 completed the initial survey, compared to 34 out of 118 (28.8%) students from the class of 2013 in the follow-up survey period. The majority of students in the initial and follow-up surveys had completed at least 4 APPE experiences at the time of survey completion (100% and 91%, respectively). Additionally, the majority of students (≥60%) in both groups had completed the required community practice and general medicine experiences at the time of survey completion. More students from the initial survey had completed an institutional practice rotation (72.4%) as compared to students from the follow-up survey (26.5%). The initial survey group and follow-up survey group had similar rates of prior pharmacy work experience (98.3% and 94.1%, respectively) with the majority of students (82.3% in initial group, 84.4% in follow-up group) working in a retail setting. In both of the study groups, hands-on activities (53.4% in initial, 55.9% in follow-up) and visual presentation (37.9% in initial, 38.2% in follow-up) were the learning styles most frequently indicated as most beneficial.

Initial Survey
In order to identify topics to remove for the spring 2012 course, student ratings from survey question number one were assessed ( Table 1). The top three topics receiving a rating of Not Useful were: Code Response (23.3%), Professionalism (21.7%), and Practice Scenarios 1 & 2 (20.0%). As a result, the Code Response and Practice Scenarios lectures were removed from the 2012 course. Although some students did not see the value in the Professionalism topic, course coordinators felt that it was a subject necessary for APPE preparation and helped to fulfill the course objective of setting "guidelines and expectations." In terms of lectures needing revision, student responses of Useful, Needs Improvement from question number one were identified. The top three topics receiving this rating were: PharmD Project Proposal/Abstract/Manuscript (45.0%), Medication Therapy Management (MTM) (43.3%), and Curriculum Vitae (CV) Tune-Up (38.3%). In response to the survey results, all three of these topics were modified; the PharmD Project and Curriculum Vitae topics were revised within the course, while the MTM lecture was moved to a community pharmacy seminar series taken during the students' APPE year. In delivering these revised topics, faculty were encouraged to keep in mind that the majority of pharmacy students felt they learned best with hands-on and visual experiences in the classroom setting.
When looking for new topics to incorporate into the spring 2012 course, student responses to survey question number two were assessed. The top three responses to this question were: evaluating information from patient profile (61.7%), locating/collecting literature & guidelines (56.7%), and understanding insurance & prescription drug coverage (46.7%). As a result, a Rounds and Patient Work-Up topic was added to teach students how to evaluate a patient profile prior to participating in one of many types of rounding experiences. Additionally, a Medicare, Medicaid, and Insurance topic was added to address deficiencies in understanding prescription drug coverage. The topic of identifying literature and guidelines was not added to the course, as this was felt to be better addressed within the Advanced Drug Information course as well as within other activities being incorporated throughout the curriculum. Finally, based on student free response comments to question number two, and Interprofessionalism topic was added to the course to help students understand the role and relationship of other healthcare professional roles within a patient care team.

Follow-Up Survey
The two topics that underwent revisions for the 2012 course both showed improvements in their ratings in the follow-up surveys ( Table 1). The overall responses for the usefulness of the CV topic were significantly changed from 2011 to 2012 (p = 0.032). A statistically significant change was also seen when looking at only Useful As Is ratings for this topic (p = 0.022). In fact, the Curriculum Vitae topic received the most positive overall feedback of the entire course with Dy, E.A., & Nisly, S.A. Journal of the Scholarship of Teaching and Learning, Vol. 14, No. 3, August 2014. josotl.indiana.edu 40 82.4% of students rating it as Useful As Is. This change was likely due the development of the CV topic into a more interactive, hands-on experience. Students were asked to participate in the lecture by providing examples of relevant experiences to include on a pharmacy student CV. Additionally, students were required to submit a CV to both a peer reviewer and a faculty reviewer in order to gain feedback. The overall ratings for the 2012 PharmD project discussion were not significantly changed from 2011 (p = 0.085); however, there was a statistically significant increase in the number of Useful As Is responses in 2012 (p = 0.024). The PharmD project discussions were, as suggested in student comments from the free response section, dispersed in small 20 to 30 minute discussions throughout the semester. The hope was to guide students through the natural progression of a research project and to allow students time to develop ideas about their own project development.
The three topics that were added to the course based on initial survey results, Rounds and Patient Work-Up, Medicare/Medicaid/Insurance, and Interprofessionalism, had mixed reviews in the follow-up survey. 41.2% of students rated the Medicare, Medicaid, and Insurance lecture as Useful, Needs Improvement. One student in particular commented that this lecture "would have been better received if the students were engaged more." Students were almost evenly split between the three ratings in terms of the Interprofessionalism lecture (35.3 % Not Useful, 32.4% Useful, Needs Improvement, and 32.4% Useful As Is). The Rounds and Patient Work-Up lecture was rated by the majority of students (58.8%) as Useful As Is.

Discussion
In this study, follow-up surveys indicated improvements in perceived student values of the topics that underwent revision in the 2012 course. This indicates that student perceptions of what is needed for clinical rotation preparation may be able to assist in creating a more effective course and may ultimately better provide students with skills to face the challenges of real-life clinical practice. New topics that were added to the course, however, had mixed reviews. The Medicare, Medicaid, and Insurance topic was rated by the many students as Useful, Needs Improvement. Additionally, although students from the initial survey were interested in learning about interprofessional relationships, the students in the follow-up survey did not find this topic to be useful in the way it was presented. This data indicate that although students are interested in how to interact with colleagues from different health profession disciplines, there may potentially be a more beneficial way to provide insight into this topic. Strategies to accomplish this in the future might include addition of a video demonstrating effective interactions amongst various health professionals, incorporation of an in-class panel of professionals who work in interprofessional settings, or even required shadowing experiences with a member of a working multidisciplinary team.
One strength of this study is the unique study design that assessed student perceptions in order to shape a preparation course for clinical experiences. Students had completed at least 4 rotation experiences prior to completing the survey, indicating a true understanding of the types of skills and information needed for the clinical rotation year. Additionally, this study identified a need for the incorporation of more hands-on experiences within the classroom setting in order to prepare students for the clinical experiences outside the classroom. The incorporation of this approach seemed to yield more favorable results during the follow-up survey period.
Although the need for hands-on experiences within the classroom setting was identified and applied in this pharmacy focused study, it appears that the same teaching approach may provide benefits across a wide variety of health profession school settings. As was the case in this study, other health profession schools have evaluated the benefits of using of hands-on learning prior to student clinical experiences. For example, Ohtake and colleagues (2013) from the University of Buffalo evaluated the impact of a hands-on intensive care unit (ICU) simulation experience on physical therapy (PT) student confidence in caring for an ICU patient and student satisfaction with this learning method. The survey results revealed that PT student confidence increased across all areas, including the technical, behavioral, and cognitive skills required to care for an ICU patient. Additionally, the majority (98%) of PT students agreed or strongly agreed that the simulation was a valuable experience that reinforced classroom content and they preferred the use of simulation experiences in additional courses.
Additionally, there are numerous studies evaluating the use of hands-on learning in nursing classes prior to clinical experiences (Bearnson, 2005;Comer, 2005). Bearnson and Wiker (2005) from Brigham Young University evaluated the use of a computer-operated mannequin called human patient simulators (HPS) in nursing student classes to add hands-on experiences. The mean student survey responses following the use of the HPS experience revealed that students agreed that HPS increased their knowledge and confidence. Comer (2005) from Purdue University North Central also used the concept of HPS to create a cost effective alternative of clinical simulation role-playing exercises for exam topics with typically high failure rates in a critical care course. Following the incorporation of these hands-on simulation experiences, student failure rates on these exams decreased by 50%. Informal student feedback Dy, E.A., & Nisly, S.A. Journal of the Scholarship of Teaching and Learning, Vol. 14, No. 3, August 2014. josotl.indiana.edu 42 following the exercises also showed that the majority of students found the hands-on experiences to be a more effective way to learn nursing concepts within the classroom. These examples from other health profession schools demonstrate the importance of hands-on learning prior to working with patients, and they support the key learning methods identified in this RX500 study.
One of the biggest weaknesses of the study is the poor survey response during each phase of the study. The initial survey response rate was 50%, while the follow-up survey yielded only a 28.8% response rate. Both of these rates are below the desired 60% response rate goal for most survey researchers. With the response rates seen, there is potential for selection bias of those individuals who had mostly positive or mostly negative perceptions of the course. Finally, there are likely differences in curriculum between the two classes studied. More structured placement of students into hospital/institutional Introductory Pharmacy Practice Experiences (IPPE) was introduced for the PharmD class of 2013. This additional experience may introduce differences in the baseline knowledge and understanding of what is needed for the APPE year.
When evaluating existing literature across all health professional schools, it becomes apparent that utilizing student feedback to shape a specific didactic course and its content is a somewhat novel idea within health science education. This study successfully achieved that goal by utilizing data from an initial student survey to identify perceived areas for improvement in RX500's ability to prepare pharmacy students for the APPE year. Follow-up surveys indicated improvements in the perceived values of the revised topics, but COPHS faculty can continue to work on delivery of newer topics being added to the course. Incorporation of hands-on teaching and learning is crucial in order to prepare health profession students prior to beginning clinical experiences.