Data Driven Approach to Analyze Competency Teaching in an Undergraduate Biology Program: A Case Study

: Recent calls to reform undergraduate biology education, including Vision and Change in Undergraduate Biology Education , have led biology departments to examine their curriculum to determine the extent to which it aligns with Vision and Change content and competency recommendations. The recently released BioSkills Guide translates the broad Vision and Change core competencies into more specific program-level learning outcomes (LOs). A curriculum map may be used to evaluate a curriculum by examining where content and skills are taught. This map can then be used to determine how well the curriculum that is actually taught aligns with the planned curriculum (a set of LOs). The Western Washington University Biology Department used a new curriculum mapping tool, the BioSkills Curriculum Survey, to examine the extent to which Vision and Change core competencies and BioSkills LOs were taught in our courses. Instructors completed the survey for every course they taught in the last two years, enabling us to gather data on competency and LO coverage and assessment across our curriculum. We answered questions about where in the curriculum competencies and LOs were taught, how different instructors teaching the same course taught LOs, the extent to which different LOs were assessed, and how teaching LOs differed in different degree tracks and different course levels. For a subset of courses, students also completed a modified survey to determine how students’ perceptions of skills coverage matched instructor’s perceptions. One main finding was that we taught Science and Society LOs less than others and, when taught, we did not often assess them. We also found that students’ perceptions of competency teaching did not align well with instructors’ perceptions. The data were used to make informed decisions about ongoing curriculum revisions. This paper illustrates the questions that can be answered using this mapping tool for competencies and we offer recommendations about how a department can take a data-driven approach to curriculum reform.


Introduction
Biology in the 21 st century is characterized by a broad array of subdisciplines each with a rapidly expanding knowledge-base, made possible by powerful new technologies (National Research Council, 2009).These recent and rapid developments have prompted discussions about what undergraduate biology students should know and be able to do when they graduate.Unlike fields such as chemistry and engineering which have standards developed by accreditation boards that can inform curricular decisions, a clear consensus about an undergraduate biology curriculum has been elusive.A survey conducted by Cheesman et al. (2007) found that biology departments offered a broad array of often specialized courses and there was little agreement about which courses should be required for all students.In response to this, a collaboration between the National Science Foundation and the American Association of the Advancement of Science (AAAS) resulted in the release of Vision and Change in Undergraduate Biology Education: A Call to Action (AAAS, 2011), which offers broad recommendations about content (core concepts that all undergraduates should understand) and competencies (sets of skills linked to disciplinary practice; Vision and Change Table 2.1) necessary for biology graduates to succeed in the 21 st century.
With the release of Vision and Change, Biology departments can critically analyze their programs to determine the extent to which their curriculum aligns with the recommendations.However, the concept statements and competencies outlined in Vision and Change are very broad and therefore provide a department with little specificity while reviewing its curriculum.This lack of specificity has recently been addressed (Branchaw et al., 2020).Perhaps the broadest response has been from the Partnership for Undergraduate Life Sciences Education (PULSE) network, which developed rubrics for departments to use in assessing five aspects of Vision and Change, including curriculum alignment (Brancaccio-Taras et al., 2016).Brownell et al. (2014) developed the BioCore Guide, which is a set of general principles and specific statements that flesh out the broad Vision and Change core concepts.In a similar vein, Cary and Branchaw (2017) developed the Conceptual Elements Framework, which provides a detailed list of principles supporting the Vision and Change core concepts and complements the BioCore Guide.To address the Vision and Change core competencies, Clemmons et al. (2020) developed the BioSkills Guide, which is a set of learning outcomes (LOs) that elaborate the broad Vision and Change competencies.For example, for the Vision and Change competency "Ability to apply the process of science," one of the BioSkills LOs is "Scientific Thinking -Explain how science generates knowledge of the natural world."While these guides, both content-focused and competency-focused, represent a significant step forward in delineating what a biology graduate should know and be able to do at the end of their undergraduate degree, a department is still left with the task of evaluating its curriculum to determine whether program-level content/competency statements and LOs are taught in classes taken by their students and assessing whether students have mastered the LOs as they progress through the major.

Curriculum Evaluation
A curriculum map may be used to evaluate a curriculum by examining where content and skills are taught.This map can then be used to determine how well the curriculum that is actually taught aligns with the planned curriculum (a set of LOs).Curriculum maps can serve several purposes including identifying curricular gaps, determining the progression of LOs throughout a program, and mapping assessment of LOs (Lam and Tsui, 2013;Joyner, 2016;Rawle et al., 2017).Curriculum evaluation is initiated by examining coursework to identify courses in which specific LOs are taught.This typically entails individual instructors reporting on each of their classes and organizing the data into a matrix that aligns courses with LO (Allen, 2004;Joyner, 2016).Organizing the data in a matrix allows curriculum alignment to be visualized in a single image.Gathering course-level data can be accomplished using instructor surveys or by groups of faculty members who teach related classes meeting to discuss LOs in their respective courses.This initial process of gathering course-level data can be cumbersome if there are many courses in the program and if there is a large number of LOs, as is often the case at the programmatic level.Developing faculty surveys can also be time consuming (Joyner, 2016).In addition, analyzing a matrix with a large amount of data (e.g.data from many courses involving many LOs) can be daunting.Curriculum mapping tools, particularly those that are Donovan, Clemmons, and Crowe Journal of the Scholarship of Teaching and Learning, Vol.22, No. 4, December 2022.josotl.indiana.eduadministered online, can streamline data acquisition and analysis (Ervin et al., 2013).Mapping tools can also help make the process more systematic, allowing for comparisons between campuses and for tracking curricular changes over time (Clemmons et al., 2022).

Applying a Curriculum Survey to Analyze Competency Teaching
The focus of this study was to map the Vision and Change competencies and associated BioSkills LOs across the curriculum of a general biology program.We used the BioSkills Curriculum Survey (Clemmons et al., 2022), a curriculum mapping tool based on the BioSkills Guide (Clemmons et al., 2020).Briefly, the BioSkills Guide proposes program-level LOs for each of the six Vision and Change competencies: Process of Science ( 6 A full list of the Vision and Change competencies and the associated BioSkills program-level LOs can be found in Appendix 1.The BioSkills Curriculum Survey is administered online and gathers information from instructors about how frequently they teach program-level LOs in each of their classes and, if the LO is taught, whether and how it is assessed (an example of a survey question can be found in Appendix 2).Thus, if the majority of instructors within a department participate, the survey provides data about the teaching frequency and assessment of program-level competency LOs across a curriculum.In addition, since the LOs are directly linked to a specific Vision and Change competency, the survey also provides data about the teaching frequencies of the competencies.For a more complete description of the BioSkills Curriculum Survey, see Clemmons et al. (2022).
Using a case study approach, we analyzed how one biology department used the BioSkills Curriculum Survey to map the teaching frequency and assessment of Vision and Change core competencies within its curriculum.We illustrate the questions that can be answered using this mapping tool and offer recommendations about how a department can take a data-driven approach to curriculum reform.

Context and Survey Administration
Western Washington University (WWU) is a primarily undergraduate institution that places high priority on research-based instruction.At the time of this study, there were 606 majors and pre-majors in the Biology Department, making it the largest department in the WWU College of Science and Engineering.There were 24 tenured/tenure-track (T/TT) faculty members, 6 senior instructors, and ~5 non tenure-track instructors.The first author (D.A.D) is a tenured member of the faculty and chair of the assessment committee.
The release of Vision and Change caused the Biology faculty to question how our curriculum aligned with the recommendations set forth in that document, and we decided to evaluate our existing courses as part of the annual programmatic assessment mandated by our university.Initially, we focused on evaluating concepts covered in our curriculum using the BioCore Guide (Brownell et al., 2014) to determine whether or not our existing courses aligned with the Vision and Change core concepts.In addition, the development and release of GenBio-MAPS (Couch et al., 2019) allowed us to assess student conceptual understanding of biology at the programmatic level.
Recently our focus turned to evaluating our existing courses to determine the extent to which Vision and Change competencies were being addressed.Our department was one of five institutions that piloted the BioSkills Curriculum Survey during its development (Clemmons et al., 2022).We subsequently took this opportunity to analyze the survey data for our annual programmatic assessment.Faculty members were asked to complete the BioSkills Curriculum Survey for every class they had taught within the past two years.They were asked about frequency of teaching LOs and whether LOs were assessed.Only the T/TT and senior instructors who taught courses taken by biology majors were asked to complete the survey (28 instructors), although two non-tenure-track instructors who taught majors courses volunteered to participate.Altogether, 27 instructors completed the survey, providing data on 31 different courses, for a total of 55 unique instructor-course combinations (we will refer to these combinations as "observations").We gathered data from multiple instructors for all of the required courses taken by our majors (200-level core courses and 300-level breadth courses).For our 400-level depth courses, each usually taught by only one instructor, we gathered data from 19 of 22 regularly scheduled courses.
To determine if students' perceptions about competency coverage aligned with instructors' perceptions, 10 of the instructors who completed the survey agreed to ask their students to complete a slightly modified survey, in which students were only asked about frequency of coverage and not about assessment.Students were offered a small amount of extra credit to complete the survey, and response rates ranged from 66.7 to 100% (mean 88.1 ± 11.2%) for these 10 courses.

Data and Analyses
We used the BioSkills Curriculum Survey to answer the following research questions (RQs): RQ 1. Are there strengths and/or gaps in coverage of Vision and Change core competencies and BioSkills learning outcomes in courses across our curriculum?RQ 2. In which courses are different BioSkills learning outcomes taught and assessed?How are they assessed?RQ 3. How does the frequency of teaching BioSkills learning outcomes differ among multiple offerings of a single course?RQ 4. How does the frequency that students are taught different BioSkills learning outcomes change from lower-level to upper-level courses?RQ 5. Do students in two different degree tracks have different exposure to BioSkills learning outcomes?RQ 6.How do students' perceptions of BioSkills learning outcomes coverage in a course compared to their instructor's perception?
Data on frequency of teaching BioSkills LOs were gathered using a 6-point Likert scale.Instructors and students were asked "How frequently is [LO] taught in this course?"with the options "not taught", "one class session", "a few class sessions", "about half of class sessions", "most class sessions", or "almost every class session."If an instructor indicated that they taught a LO, they were asked if that LO was assessed ("yes" or "no") and, if so, how it was assessed (open-ended text box).In addition, for a random subset of LOs, instructors were asked open-ended questions to determine how they assigned a frequency for that LO to determine whether instructors were using valid response processes (Behr et al. 2017).Throughout this paper, we refer to the BioSkills Guide program-level LOs by their shorthand names (see Appendix 1 for shorthand names and full text of each LO), however instructors and students were shown the full text of the LO when answering survey questions.An example of the survey question for one of the LOs can be found in Appendix 2 (questions for the other LOs were identical except for the insertion of other LO text).To analyze instructor responses to the survey (RQs 1-5), we visualized data as heatmaps or stacked bar charts.Heatmaps (always colored green) were used when the analysis required knowledge of specific courses or course sections in which the LO was taught.Stacked bar charts (always colored blue) were used when the analysis only required data at the programmatic level.For both types of visualization, we averaged frequency responses across multiple instructors who reported on the same course to ensure that courses taught by multiple instructors were not overrepresented in the visualizations.To calculate these means from the Likert scale responses, we treated the response scale as numeric (0 = not taught, 5 = almost every class session) and rounded to the nearest integer.We chose to use mean rather than median with our relatively small sample size (3-6 instructors for courses taught by multiple people) since means would allow us to better observe slight differences in distributions between responses for different LOs.Two RQs required data from individual observations, so we did not average frequency responses.Specifically, for RQ 2 we visualized teaching and assessment for each instructor-course combination and for RQ 3 we purposefully compared multiple offerings of the same courses.For competency-level analyses that had nested LO-level analyses (RQ 1), we aggregated responses for all LOs associated with each Vision and Change competency.To determine the most and least frequently taught LOs (RQ 1), we compared mean frequency responses across all courses and we averaged frequency responses across multiple instructors who reported on the same course.
When average student experience was part of the analysis, we weighted instructor response frequencies by annual student enrollment so that large-enrollment courses that impacted many students contributed more to the visualization.For analyses in which program-level data were visualized with stacked bar charts (RQs 4 and 5), mean instructor response frequencies were weighted by annual student enrollment.For analyses in which course-level data were visualized with heat maps (RQs 3-5), each cell represented a course and we scaled the heatmap Y-axis so that it reflected relative annual student enrollment of each course.We estimated annual student enrollment by summing the number of students enrolled in each course over an academic year, averaging across the previous two academic years to account for any irregularities and to calculate enrollment for courses that are not offered every year.We assigned elective courses to particular degree tracks (RQ 5) based on student degree requirements for each track, however all courses can be taken by students in other tracks as electives.
To analyze student survey data (RQ 6), we visualized the distribution of student responses as well as the mean student frequency response (treating the response scale as linear) for each LO in each course relative to the instructor's response.
Following survey administration and data analysis, the results were presented to the Biology faculty at a department meeting as part of our yearly programmatic assessment and the results were discussed.To gather further insight about how individual faculty members planned to use the information and how people thought the information could be used at the departmental level, faculty members were asked to respond to a short Qualtrics survey after the faculty meeting.Three questions were asked: 1) How might we use the information from the BioSkills survey at the program level?, 2) How might you use the information from the BioSkills survey at an individual level?, and 3) Is there any other information you would like that I may be able to provide from the survey results?Eleven faculty members responded, and the information gathered from this survey supplemented our analyses of the six RQs.This study was approved by the University of Washington, Human Subjects Division as exempt (STUDY00001746), as part of a larger multi-institutional study.

BioSkills Curriculum Survey Results and Implications for Curriculum Renovation
RQ 1. Are there strengths and/or gaps in coverage of Vision and Change core competencies and BioSkills learning outcomes in courses across our curriculum?Although all competencies were reported taught in at least one class session in the majority of courses (Figure 1A), there were large differences in the extent to which competencies were taught.The Biology faculty taught Quantitative Reasoning and Process of Science LOs to greater extents than the other competencies.Instructors reported that they taught one or more Quantitative Reasoning LOs for at least a few sessions during the quarter in 82% of courses and they were covered in at least half of class sessions in 46% of courses.Coverage of Process of Science LOs was very similar to Quantitative Reasoning LOs (Fig. 1A).Science & Society was the core competency taught least in our curriculum.Instructors reported that they taught Science & Society LOs during at least a few class sessions in 61% of courses, however it was taught in half or more class sessions in only 14% of courses.Modeling was also underrepresented in our curriculum (Figure 1A).By breaking down the broad Vision and Change core competencies into more specific BioSkills LOs, we obtained greater detail about which aspects of core competencies faculty emphasized in their teaching.The frequencies at which different BioSkills LOs were taught differed across our curriculum (Figure 1B).Some LOs were taught during at least a few class sessions in nearly every course, including Data Interpretation & Evaluation (100% of courses), Connecting Scientific Knowledge (94%), and Scientific Thinking (91%).In contrast, Interdisciplinary Problem Solving was taught during at least a few class sessions in only 42% of courses.It is not necessary that all LOs be taught with the same frequency.Indeed, it is not surprising that Data Interpretation & Evaluation was taught to some extent in all of our courses since it is a basic skill used in all aspects of science.Higherlevel LOs such as Interdisciplinary Problem Solving might be taught in upper-level or capstone courses and would thus be taught less frequently over the entire curriculum.Curriculum mapping allowed the Biology faculty to see where LOs were taught and to discuss whether there were LOs that should be taught more frequently.
Within a competency, there was often one LO that was taught with less frequency than the other LOs.For example, within the Communication & Collaboration core competency, three of the LOs (Communication, Collaboration, and Metacognition) were taught with higher frequencies than Collegial Review (Figure 1B).A similar trend was seen in Process of Science and Interdisciplinary Nature of Science.Within all three of these competencies, one or more LOs were in the top six most frequently taught LOs, while other LOs within the competency were in the least frequently taught LOs (Table 1).For example, Process of Science, which has six LOs (Appendix 1), had three LOs (Data Interpretation & Evaluation, Scientific Thinking, and Question Formulation) in the six most frequently taught skills while one LO (Doing Research) was in the six least frequently taught skills.This is not surprising given that doing research is a higher-level, more time-intensive skill, often performed in upper-level courses, and it builds on the other LOs within the Process of Science competency.Two of the three LOs in the Science and Society competency (Societal Influences and Ethics) were among our six least frequently taught LOs, which gives us insight into why Science and Society was our least covered core competency.Identifying competencies and LOs that faculty members recognize as undertaught can lead to curricular changes by individual instructors.Perhaps one of the most straightforward changes that can be made by an instructor is the deliberate inclusion and assessment of undertaught skills in their courses.The results of our curriculum evaluation prompted several WWU instructors to consider how they could more intentionally include skills within the Science & Society and Modeling core competencies, as these were identified as underrepresented in our department's offerings.As one Knowing what is not being covered/assessed in different classes can allow instructors to be more mindful of what topics could be strengthened to provide students in specific disciplines the skills the department feels are important for all of our majors.
For example, although there were 48 cases where the LO Science's Impact on Society was reported as being taught, 35 (73%) indicated teaching in either one class session or a few class sessions.By reading embedded web probing responses in the BioSkills Curriculum Mapping Survey, we learned that this LO was often addressed in class on an impromptu basis and not as a planned component of the course curriculum.Instructor responses included: I think that we have these discussions a few times during the quarter, but they are not planned into the curriculum.
There are a few topics in Bio [2XX] (when we have the time) that lend themselves to discussions about how students use scientific information that they've heard/read about in popular media.Students will occasionally ask a question highlighting popular misconceptions about science, which will fuel further discussion.
Thus, one takeaway from our results was that if the Biology faculty thinks understanding societal connections to science is important for our majors, we need to teach these skills much more explicitly and assess them in more meaningful ways.
Modeling was another core competency that was undertaught in our curriculum, and this may be due to differing understanding of what modeling is and therefore inconsistent interpretation of the LOs across the faculty.As one instructor commented: I think that we should have a discussion about what "modeling" means and encourage instructors to point out that what the class may be doing during an exercise or discussion is actually a form of modeling.This should be a type of epistemological discussion on how we conduct science and how modeling interacts with other approaches (e.g., observation and experimentation).
Although modeling is a core scientific practice, there is little consensus about what models are and how the practice of modeling should be used in the classroom, as well as disciplinary differences in the practice of modeling (Manthey and Brewe, 2013;Svoboda and Passmore, 2013;Gouvea and Passmore, 2017).Students tend to view models as physical representations of phenomena, while experts have a more sophisticated view that includes abstract models used to understand and communicate ideas (Grosslight et al., 1991;National Research Council, 2012;Quillin and Thomas, 2015).Integrating modeling into undergraduate biology classrooms to help students learn content as well as the practice of modeling has increased in recent years, but there is ongoing discussion about how modeling can be used as a pedagogical tool (Manthey and Brewe, 2013;Diaz Eaton et al., 2019;Wilson et al., 2019).
Curriculum evaluation can also facilitate data-driven changes at the departmental level.Significant gaps in competency coverage could be tackled by developing new courses targeted to address a specific competency.For example, many science departments offer Science and Society or Ethics courses.Undertaught competencies could also be addressed by intentionally including them in courses being developed to fill content gaps.In 2014, the WWU Biology Department completed a curriculum evaluation of the Vision and Change core concepts using the BioCore guide (Brownell et al., 2014).At that time, we identified gaps in upper-level content, including physiology and developmental biology.These gaps are currently being addressed in a departmental curriculum revision and we are restructuring our core courses, thinking about both content and competencies that our students require.Both these efforts will be informed by knowledge of which courses cover different core competencies.
RQ 2. In which courses are different BioSkills learning outcomes taught and assessed?How are they assessed?
If a LO was reported taught in a course, it was likely to be assessed (Figure 2).Some LOs were assessed in nearly every course in which they were taught (e.g.Data Interpretation & Evaluation and Quantitative & Computational Data Analysis).Other LOs were almost never assessed, even though they were reported taught in many courses (e.g.all of the Science & Society LOs as well as Metacognition).

Figure 2. Teaching and assessment (both formative and summative) of BioSkills learning outcomes in individual courses across the curriculum.
To investigate whether infrequently taught LOs (those reported taught once or only a few times) accounted for most of the unassessed LOs, we compared the total number of reports of infrequently taught LOs that were assessed to those that were not assessed.Surprisingly, infrequently taught LOs that were assessed comprised 55% of infrequently taught LOs, indicating that instructors more often than not assessed LOs they only taught once or a few times during a class.The three LOs within Science & Society were reported taught but not assessed more than any other LOs.For example, Ethics was reported taught in one or more class sessions in 40 out of 59 observations, yet of those 40 observations in which it was reported taught, instructors reported assessing it only five times.LOs reported taught in half or more class sessions were very likely to be assessed (88% of all reports), although this means that 12% of reports of frequently taught LOs included no assessment.These results indicate a need to more closely align assessment with skills taught in courses.
The open-ended question on the BioSkills Curriculum Survey about how LOs were assessed allowed us to identify assessments for LOs that were taught by a majority of instructors but not commonly assessed.For example, instructors who both taught and assessed Metacognition reported using exam wrappers, exit slips, quick writes, response to discussion board prompts, questions at the end of worksheets, and end-of-course surveys to prompt and evaluate students' use of Metacognition.Some instructors expressed uncertainty about how to assess some skills, which prompted other instructors to share assessment techniques.Sharing these strategies should encourage faculty members to try alternative types of assessment in their classes.
Providing feedback to students about content and skills mastery is an important aspect of learning (Black and Wiliam, 1998).Thus, explicitly assessing LOs that instructors report teaching but don't report assessing is an important change that could be made relatively easily.Our assessment results indicated that instructors were very likely to assess easily quantified LOs (e.g.Data Interpretation & Evaluation and Numeracy) while LOs that were more subjective (e.g.Metacognition and Ethics) were less likely to be assessed, even though they were reportedly taught in class.As one instructor commented on the survey following the faculty meeting: We need to think about how to consistently assess the things we're not assessing (metacognition, for example), and we need to do a better job on the science and society interface.That'll help our discipline seem more relevant to peoples' lives.

RQ 3. How does the frequency of teaching BioSkills learning outcomes differ among multiple offerings of a single course?
For courses taught by multiple instructors, we anticipated seeing variation in reported frequency of competency teaching.Course sections of Biol 2XX exhibited variability across all core competencies (Figure 3A), despite the course being developed by a faculty working group around a common set of core ideas and content LOs.All instructors teaching the class had collaborated on developing the curriculum, although the development focus for that working group was on content and not on competencies.In Biol 3XX, teaching of Process of Science, Quantitative Reasoning, and Science & Society were relatively uniform, while Modeling and Communication & Collaboration were more variable (Figure 3B).While some of this variability was likely due to interpretation of BioSkills LOs and the associated survey questions, it also reflected different approaches instructors are taking to teach common curricula.For example, an instructor who emphasizes group work in class would increase the use of Communication and Collaboration without changing course content, while an instructor who focuses on data and figure analysis to teach content would emphasize different LOs within Process of Science and Quantitative Reasoning.
Results of curriculum evaluation could encourage conversations between instructors to coordinate competencies taught in different sections of the same course.At WWU, we have done this for content covered in the 200-level core courses, and some 300-level courses, by developing course LOs and instructional sequences.We have not had the same level of course-specific discussion about competency coverage, although a working group (separate from the content working groups) developed a set of instructional sequences for the Vision and Change competencies that spanned the 200-level core courses.Although these sequences were never fully enacted, this earlier work would facilitate writing course-level competency LOs that would elevate competency coverage and promote conversation.As one instructor commented on the survey following the faculty meeting: As far as I know we are currently approaching BioSkills passively at the program level.That is, individual instructors may address specific BioSkills but nothing is planned or consistent from instructor to instructor.It seems like we could use the BioSkills data to make decisions about which courses should/could cover which skills.
In short, writing course-specific LOs associated with competencies would increase consistency between course offerings, which would in turn enhance scaffolding between lower-level and upperlevel courses.It would also allow instructors to share strategies for teaching different LOs, although individual instructors' teaching styles would still influence competency coverage.

RQ 4. How does the frequency that students are taught different BioSkills learning outcomes change from lower-level to upper-level courses?
Across all competencies, the frequencies at which BioSkills LOs were reported taught were greater in upper-level courses (300-and 400-level) compared to lower-level (200-level) courses (Figure 4).Teaching frequencies of Process of Science and Quantitative Reasoning LOs increased progressively from 200-to 400-level courses, while frequencies of Modeling and Science & Society LOs were relatively consistent between 300-and 400-level courses.These results led us to consider how competencies and LOs might be scaffolded between lower-level and upper-level courses.As one instructor commented on the survey following the faculty meeting, the department could use the BioSkills data by: • Looking to fill in gaps in key skills.
• Discussing how to build skills over the course of the program (consciously developing activities across courses that sequentially build on skills).• Developing learning progressions across courses (building on those developed for the 200-series) to help inform the above two activities.
Although we found that LOs were taught with greater frequency in upper-level courses compared to lower-level courses, a more detailed analysis could be done on sequences of courses [e.g.Introduction to Cellular and Molecular Biology (200-level)  Cellular and Molecular Biology (300-level), Methods in Molecular Biology (300-level lab)  400-level Cell and/or Molecular Biology electives] to determine how LOs are scaffolded between courses.This would be similar to curriculum mapping methods that ask instructors to rank whether a skill is introduced, reinforced, or mastered as a means of determining how competencies are scaffolded across classes.Recent documents at both the undergraduate and K-12 levels have emphasized the importance of teaching both content and science practices [e.g.Vision and Change (AAAS, 2011) and the Next Generation Science Standards (NGSS) (NRC, 2013)], leading to discussion about how competencies could be scaffolded across grade bands or lower-and upper-level courses.The NGSS provide learning trajectories across grade bands for the eight Science and Engineering Practices identified in that document.Vision and Change offers recommendations about introducing competencies early in a student's major and including competencies in every course in the program.However, there is little guidance about how students should progress through competencies.Content complexities and educational contexts of teaching and learning practices make developing learning progressions for individual competencies difficult (Pierson et al., 2019).It might therefore be up to individual departments to decide how competencies should be progressively taught in their curriculum.These decisions would likely be influenced by the size of the department and the relative sizes of lowerlevel and upper-level classes.Large class sizes in introductory courses, which is often the case at universities, might limit an instructor's ability to teach and assess competencies.It is also important for universities to communicate clearly with local community colleges to support transfer students in the transition between lower-level and upper-level courses, so that progression through competencies will be facilitated for transfer students.

RQ 5. Do students in two different degree tracks have different exposure to BioSkills learning outcomes?
Students in the Molecular and Cellular Biology (MCB) track generally had more exposure to the BioSkills LOs compared to students in the Evolution, Ecology, and Organismal Biology (EEO) track.Although all students were taught LOs in the core courses required of all students, a greater proportion of elective MCB courses addressed LOs in most or almost every class session compared to elective EEO courses (Figure 5A).For example, elective MCB courses covered LOs within Interdisciplinary Nature of Science and Process of Science to a greater extent than EEO courses.Although MCB and EEO were not the only two tracks in the department (there were five in the Biology BS), we chose them to illustrate how tracks could be compared because they included the largest proportions of students in our department (49.7% between the two tracks), and they had the most differences in courses that students might take.
Individual elective courses within the two tracks differed in the extent they covered LOs (Figure 5B).This result led instructors to consider incorporating LOs into their upper-level courses and also prompted discussion about more intentional articulation between lower-level and upper-level courses.This may be particularly important for competencies that have LOs that build upon each other.For example, the Process of Science competency has LOs that are linked (Question Formation, Study Design, Data Interpretation & Evaluation, Doing Research) and should be thoughtfully integrated, requiring collaboration of instructors teaching across the curriculum.Other competencies, such as Communication & Collaboration and Science & Society, have LOs that are not as hierarchical.Intentionally scaffolding competencies throughout our curriculum should help our students master competencies.Killpack et al. (2020) found that scaffolding skills instruction within a course improves skills acquisition as well as students' perception of scientific ability.In addition, scaffolding skills within a course can also help students succeed in future courses, both in content and skills acquisition (Dirks and Cunningham, 2006;Coil et al., 2010).

RQ 6. How do students' perceptions of BioSkills learning outcome coverage in a course compare to their instructor's perception?
Of the ten courses for which we have data, some had very good agreement between students' perceptions of LO coverage and their instructor's perception, but others did not.Figure 6 provides data from a single course as an example of the information gained to answer this question.We kept this analysis at the individual course level to provide feedback to individual instructors about their students' perceptions of LO coverage.Comparing the student mean response (teal circles) to instructor response (orange circles) for this course, we see that student mean response was within one response category of the instructor's response for 15 of the 20 LOs (this was the highest agreement among the 10 courses).For four of the LOs (Doing Research, Study Design, Ethics, and Interdisciplinary Problem Solving), students perceived the LO being taught during at least a few class sessions while the instructor did not report teaching it.Over the entire 10 courses, there was least agreement for Interdisciplinary Problem Solving, Modeling, and Scientific Thinking.There was very good agreement for Data Interpretation & Evaluation and Societal Influences.These results prompted instructors to examine how they message competency coverage to their students, since students' perceptions of LO coverage often did not align well with instructor perception.As one instructor commented on the survey following the faculty meeting: [I can use the results of the survey] to inform best teaching practices.I, for one, was really interested in the data you showed that looked at what the students thought they were learning and what the instructors thought they were teaching.I took away from that, that I should perhaps be more explicitly identifying the skills that specific exercises or assessments are helping to build.One reason for the lack of alignment may be due to instructors having a more sophisticated understanding of competencies and LOs than students.This could lead to students not recognizing when a LO is being taught, which would lead to students underreporting LO coverage, or to students thinking a LO is being taught when it actually isn't, which would lead to overreporting.For example, Modeling was a LO that had low agreement across the 10 courses and "model" has been identified as a jargon term that students struggle with (Zukswert et al., 2019).In our survey, students tended to overestimate LO coverage or to report coverage of a LO when the instructor reported it not covered.Our analyses also compared responses from all the students in a class, resulting in a range of responses, to the one instructor response.Thus, if the instructor reported the LO taught at the highest (almost every class) or lowest (not taught) frequencies, student mean responses would tend to be different than the instructor's response.Whatever the cause, instructors agreed that students may not recognize when LOs are being taught, making explicit messaging all the more important.Clemmons et al. (2022) found that students recognized LO instruction more often when LOs were included on assessments than when LOs were included on syllabi.This suggests that explicitly targeting LOs during class meetings by letting students know LOs are being taught and including LOs on assessments are important.Better messaging of LOs would improve student understanding and mastery of competencies that are important for biologists.

Broader Implications
The BioSkills curriculum survey (Clemmons et al., 2022) allows departments to map Vision and Change core competencies (AAAS, 2011), and associated BioSkills LOs (Clemmons et al., 2020), in a systematic manner across courses in a curriculum.This in turn allows departments to make data-driven decisions about curricular renovations, both at the individual course level and at the department level.Instructors can modify their courses to incorporate underrepresented competencies and LOs, can develop assessments for LOs that are taught but not assessed, and can more explicitly message LOs to their students.Instructors teaching the same course can collaborate with colleagues to share skillsbased pedagogies.Departments can more deliberately scaffold teaching competencies between lowerlevel and higher-level courses and ensure that students in different degree tracks have similar experiences with competencies.Responding to systematically collected data would likely motivate instructors to participate in these types of curricular reform.
Curriculum evaluation can also assist departments in fulfilling university assessment and accreditation requirements in a meaningful way.At WWU, each department is required to assess at least one departmental learning outcome every year, and many departments rely on instructor assignments and student work to meet assessment requirements.The use of validated, published tools has allowed our department to gather quantifiable data that can be used to track changes over time.This has led to meaningful, data-driven decisions about our curriculum.
While we have illustrated how curriculum evaluation can be used by a single department to make informed decisions about curricular reform, our survey results were similar to those of the other pilot institutions (Clemmons et al., 2022), suggesting that our recommendations are broadly applicable.For example, three of our four least commonly taught LOs (Collegial Review, Societal Influences, and Ethics) were the least commonly taught LOs across other institutions, while our most commonly taught LOs (Data Interpretation & Evaluation, Connecting Scientific Knowledge, and Scientific Thinking) were among the most frequently taught at other institutions (Figure 3 in Clemmons et al., 2022).As with instructors at other institutions, WWU instructors reported teaching across all six competencies but did not assess Communication & Collaboration or Science & Society to the same extent as the other competencies (Figure 4 in Clemmons et al., 2022).Finally, while WWU instructors LOs), Quantitative Reasoning (2 LOs), Modeling (3 LOs), Interdisciplinary Nature of Science (2 LOs), Communication & Collaboration (4 LOs), and Science & Society (3 LOs).Although the BioSkills Guide also breaks down the program-level LOs into more specific course-level LOs, the BioSkills Curriculum Survey is designed to map the program-level LOs.

Figure 1 .
Figure 1.Proportions of courses in which Vision and Change competencies and BioSkills learning outcomes were taught.A) Proportions of courses in which instructors reported teaching Vision and Change core competencies at different frequencies across major's courses (N = 33) in the WWU Biology department curriculum.To calculate the proportions of a competency, LOs within each competency were aggregated and we averaged frequency responses across multiple instructors who reported on the same course.B) Proportion of courses in which instructors reported teaching BioSkills LOs at different frequencies.Within each learning outcome, frequency responses across multiple instructors who reported on the same course were averaged.
Donovan, Clemmons, and Crowe Journal of the Scholarship of Teaching and Learning, Vol.22, No. 4, December 2022.josotl.indiana.edu Donovan, Clemmons, and Crowe Journal of the Scholarship of Teaching and Learning, Vol.22, No. 4, December 2022.josotl.indiana.eduinstructor commented on the survey following the faculty meeting where we discussed the curriculum survey results: Donovan, Clemmons, and Crowe Journal of the Scholarship of Teaching and Learning, Vol.22, No. 4, December 2022.josotl.indiana.edu Donovan, Clemmons, and Crowe Journal of the Scholarship of Teaching and Learning, Vol.22, No. 4, December 2022.josotl.indiana.edu Donovan, Clemmons, and Crowe Journal of the Scholarship of Teaching and Learning, Vol.22, No. 4, December 2022.josotl.indiana.edu

Figure 3 .
Figure 3. Frequencies of BioSkills learning outcomes reported taught by different instructors of the same course.Each row of cells are reported frequencies from an individual instructor.A) Biol 2XX is a lower-level course taken by all majors.B) Biol 3XX is an upper-level course taken by all majors.

Figure 4 .
Figure 4. Frequencies of different BioSkills learning outcomes reported taught in A) upperlevel and B) lower-level courses.Courses taught by multiple instructors are represented by average frequencies and the Y-axis is scaled relative to the average number of students in the course per year Donovan, Clemmons, and Crowe Journal of the Scholarship of Teaching and Learning, Vol.22, No. 4, December 2022.josotl.indiana.edu

Figure 5 .
Figure 5. Proportions and frequencies of elective courses teaching BioSkills learning outcomes in two different degree tracks.A) Proportions of courses taken by students in the Molecular and Cellular Biology (MCB) emphasis compared to electives taken by students in the Evolution, Ecology, and Organismal Biology (EEO) emphasis.B) Frequencies of different BioSkills learning outcomes reported taught in MCB and EEO elective courses.

Figure 6 .
Figure 6.Frequencies of BioSkills learning outcomes reported taught by students compared to reported taught by the course instructor.Grey dots are students' responses (N = 20 for this course), weighted by the number of students who chose that frequency.Teal dots are mean student response and orange dots are instructor response.

Table 1 . The six most and least frequently taught BioSkills learning outcomes in our curriculum.
See Appendix 1 for the full list of the 20 learning outcomes.