Assessing Public Health Majors through the Use of e-Portfolios

When assessing an entire academic program, there are various possibilities; most require students to reflect holistically on knowledge learned. Final presentations, internships, theses, and dissertations all require the students to recall the entirety of their learning experience. These are more traditional ways to assess the student as well as the program as a whole. However, with advancement in technology, the use of electronic portfolios (e-Portfolios) has been advocated to highlight student accomplishments as well as to document program and course outcomes. The following project illustrates the use of e-portfolios and develops specific rubrics in order to measure both student learning and program assessment. The use of e-Portfolios as an assessment measure was developed and implemented into the Public Health Program. All graduating students, upon completing their internships, create an e-Portfolio. These portfolios are used by faculty for both student and program assessment purposes. Data collected over the 7 semesters provides valuable insight into both students’ level of competencies and program outcomes for both Pubic Health core goals and objectives.

understanding, it is a way to measure the effectiveness of the course being taught and the methods of doing so; or as a wholean academic program.It holds the purpose to constantly improve instructing methods and student learning.Walvoord describes three crucial steps in assessment, which are, as follows: goals, information, and action (2010).Having goals is necessary to assessment, because both or all parties' involved need to agree on what the objective or expected outcome is.For example, one must know what skills and/or knowledge is expected to be gained from the course in order to assess.It is impossible to assess unless there is a mutual understanding of what is being learned.
Information commonly referred to as measures or evidence is the actual assessing of student learning (Walvoord, 2010).This is the empirical data that is collected by providing various types of assessments on student learning and the success of an academic program.Gathering this information is done by things such as tests, projects, papers, portfolios …etc.The final step of the assessment process is to utilize what is learned or gained from the assessments.Taking this information and communicating it with other instructors in an attempt to tailor one's courses and ways of assessing is an example of action.However, the results of your assessments are not necessarily accompanied by the answer of how to improve them.In order to do so, past and individualized research is needed.
Assessment is necessary because in most, if not all cases, assessment leads to action.In these cases, the use of assessment acts as an aid in taking these actions.It is necessary in discovering what the strengths and weaknesses are of the students being assessed.Without knowing the weaknesses, there would be no basis or idea of what to improve.Basically, it serves as an educated guess, rather than a blind assumption lacking evidentiary support.It enables the education system to constantly improve student learning.Specifically, Suskie (2009), described assessment within a course of study.More times than not, at a college or university level, there is a variety of instructors heading the same course, most of which do not have a unanimous way of assessing student learning (Suskie, 2009).This is not necessarily ideal in gathering data on assessment within a program.While trying not to discourage one's own personal flair towards teaching, there still needs to be a common foundation; for example, a commonalty among exams and assignments.However, when assessing an entire academic program, there are many different possibilities.During higher-level courses, there are many assignments that require the students to reflect on things previously learned.For example, final presentations, internships, theses, and dissertations all require the students to recall the entirety of their learning experience.This is a way to assess the student as well as the program as a whole.
A recent study conducted by the National Institute for Learning Outcomes Assessment discovered that there was an evident difference in the types of assessment depending on the academic program (Glenn, 2011).The types of assessment considered in their study were as follows; locally developed, licensure, and standardized exams; alumni and employer surveys; student interviews or focus groups; capstone courses; culminating projects or demonstrations; and portfolios.Eleven different course majors or departments were included within the study; among these was the health sciences department, which rated over fifty percent in seven of the nine categories of assessment.Concluding that out of the 982 academic programs included, over 50% of the health sciences departments stated that the majority or all of their students in the department are assessed with seven of the forms of assessment in question.In order to improve student learning, there must be constant flowing assessment, both at a course level and within an academic program.

e-Portfolios and Student Learning
An electronic portfolio or sometimes known as an eportfolio, e-portfolio, digital portfolio, or online portfolio, is a way in which a user can display a collection of electronic evidence.Such electronic evidence may include inputted text, electronic files, images, multimedia, blog entries, and hyperlinks, and demonstrations of the user's abilities and platforms for self-expression, and, if they are online, they can be maintained dynamically over time.(Zimmerman, 2012).In education, students' e-portfolios are a collection of student work that can advance learning by providing a way for them to organize, archive, and display work.Also, the electronic format of an e-portfolio, easily allows instructors to evaluate students portfolios and has become a popular alternative to paper-based portfolios because they provide the opportunity to review, communicate and give feedback in an asynchronous manner.In addition, students are able to reflect on their work, which makes the experience of creating the e-portfolio meaningful.In addition, a student e-portfolio may be shared with a prospective employer or used to record the achievement of program or course specific learning outcomes (Lorenzo & Ittelson, 2005).
Finally, e-portfolios can facilitate students' reflection on their own learning, leading to more awareness of learning strategies and needs (Mooney, 2014;van Wesel & Prop, 2008).
The use of student e-Portfolios "support of individuals through a critical reflective approach to competency development, supporting professional collaboration, and providing a structure for planning career progression" (Andre, 2010) and past research provides us with evidence of numerous positive outcomes.Specifically, the following illustrate this positive relationship:  Students feel that e-Portfolios enhance their education by assisting them in integrating academic and experiential learning (Hayward et al., 2008). e-Portfolio encourages application and integration of knowledge, helps students recognize performance gaps, fosters student development, and promotes a student's responsibility for learning (Bierer, Dannefer, Taylor, Hall, & Hull, 2008). Reflective e-portfolios can assist students in organizing and integrating academic and experiential learning and can illuminate developmental transitions (Hayward et al., 2008;Lewis & Baker, 2007). In cases where the ePortfolio had been introduced across a healthcare program, a number of learners had also begun to use feedback provided through the ePortfolio as a springboard for reflection and planning for future program development (Peacock et al., 2009;Peacock, Gordon, Murray, Morss, & Dunlop, 2010).
As indicated by these findings, e-portfolios have shown to be valuable reflective mechanisms for both student learning and academic program assessment.The current study uses students' internship e-portfolios in order to assess Public Health undergraduate student learning and program goals and objectives.This study tracks Public Health assessment data over the last four years to identify students' level of proficiency in Public Health Program Core Goals and Objectives.This information illustrates the use of e-portfolios in showcasing not only student learning from undergraduate courses, but how students apply this knowledge during their internship experience.Development of student e-portfolios provides a reflective mechanism for both students and faculty to assess student learning and ascertain quantitative program assessment data.Thus, the following research questions are posed:


Students will develop the ability to demonstrate leadership when applicable: Leadership is a process whereby an individual influences a group of individuals to achieve a common goal (Northouse, 2007).
Using these outcome expectations, the authors developed a Likert scale to be used to assess internship portfolios.In May 2010, the authors attended the Assessment Workshop at Stockton and were able to work on fine-tuning the above assessment process.The following were the changes that were made to our assessment protocol and were implemented starting fall 2010.The internship e-portfolios and final internship presentations were chosen as the most appropriate factors to assess student outcome since they are all final product and are based on student classroom and fieldwork experiences; hence, they are the best representation of the level of PUBH competence our students attain by graduation.All students are required to present their internship projects to the faculty, other interns, and students currently enrolled in PUBH 1100 Intro to Community Health.Once students complete their presentation, they submit their internship e-portfolios.Faculty review and assess these portfolios and presentations and then fill out the Likert scale assessing students' level of proficiency on each of the core and track objectives.Students are also required to complete the same Likert scale to indicate their perception of their competence on the various skill sets.
As highlighted above, past research illustrates the possible benefits and values of ePortfolio, but given the time-consuming manner of implementing such a process, it is not without challenges (Andre, 2010).The use of e-portfolios involves teaching both faculty and students how to maneuver properly through the software and students need to have ample time to learn and utilize the program properly (Moores & Parks, 2010).In addition, Moores and Parks state that supportive, clear, and succinct guidelines are also necessary for optimal ePortfolio outcomes, including clear guidelines and criteria regarding the structure and size of the students' ePortfolios.Given these findings, it is important to note that students in this study were required to take a Public Health Marketing Course prior to completing their internship.The semester long Social Marketing Campaign for this class needed to be presented using the e-portfolio format, giving students past experience with this software.In addition, all students are required to take PUBH 1100 prior to taking their internship, therefore, have had the opportunity to attend the internship presentations and see examples of student e-portfolios.Finally, students are provided an internship packet prior to starting their internship experience that includes both the core competencies and the assessment scale.Students are informed that once they have completed their internship hours, created their e-portfolio and given their presentation, they will be required to fill out the assessment scales.Specifically, students are instructed both verbally and in writing to use their e-portfolio as a way to reflect upon your experiences both in and out of the classroom and use this technology to illustrate your levels of proficiency on these core competencies.
Formal collection of data began with the Public Health Internship class of Fall 2010, and has been collected every semester since.This report includes data analysis from then until Fall 2013.Authors entered quantitative data into an SPSS database for evaluation and created a Word file for all qualitative data.The following section reports descriptive statistics for both student and faculty assessment of the 13 core competencies, along with a Needs and Solution Chart based on the qualitative data.

Results
Goal Students' oral communication was rated by both faculty and students similarly with a range of in the high 7 to low 8 range.Students' ability to communicate using electronic methods was a bit higher than their mere ability to demonstrate effective communication; faculty and students were again comparableboth with a mean of 8, with student ratings just a bit higher than faculty.These results appear to have predictive validity given that students should be a better assessor of their own electronic ability (usage of such medias is usually higher than that of faculty).Students' written communication reveals a bit more discrepancy between faculty and student, faculty (mean = 7.7) over a point lower than students' assessment (8.8).This again appears to indicate predictive validity, as faculty tends to be more rigorous with levels of writing proficiency.Summary: Both faculty and students' assessment of the above objectives were comparable, with students tending to be slightly higher; average score for all three objectives are in the 80% range.

Goal
Qualitative Data: After students rate their proficiency on the objectives, they were asked to provide suggestions/comments about their experience as a Public Health Major.Specifically elements they liked, what things should be changed / added, what was beneficial to their learning, and what was not, and how these changes would improve the major.Based on the review of qualitative data over the last 3 years, PUBH faculty has identified some areas that need improvement / change.Based on this data, the following needs have been identified and possible solutions discussed and implemented: NEED

Discussion
With the advancement in technology, the use of electronic portfolios (e-Portfolios) has been advocated to highlight student accomplishments as well as to document program and course outcomes.For example, Perks and Gallantino (2013) found that the development and implementation of an ePortfolio and grading rubric has provided a framework for assessment of student learning for a Holistic Health Minor.Specifically, results indicate that the use of eportfolios allows students to incorporate greater reflection earlier in the process for timely feedback from each course to enhance learner engagement, while the use of a rubric for assessment provides rich insight into program outcomes.The current research study illustrates the use of e-portfolios and developed specific rubric in order to measure both student learning and program assessment.The use of e-Portfolios as an assessment measure was developed and implemented into the Public Health Program for three consecutive years.During this time, both faculty and students provided quantitative and qualitative data in order to measure student learning and program assessment.
Student and faculty data obtained on students' level of proficiencies on 13 specific competencies used to operationalize three more general program goals, indicate that students' josotl.indiana.edulevel for proficiency fall within the moderate to high range for the following 13 variables: oral communication; written communication, communicating using electronic methods; fundamental public health principles; public health research, computational analysis; epidemiological methodology; understanding and implementing marketing principles; general public health knowledge; leadership; critical thinking; and cultural competency.In addition, faculty and students are assessing students' level of proficiency approximately the same, with faculty consistently assessing students slightly lower.Results also provide useful information with regards to what proficiencies are ranked slightly lower than others.For example, more skillbased competencies such as writing, oral communication and mathematics (research and epidemiological methods and statistics) are ranked slightly lower than other more theoretical competencies such as fundamental principles, general public health knowledge, and critical thinking.This type of feedback can help guide future program and course curriculum, along with teaching strategies and techniques.
Qualitative data obtained from students' assessment measure both supported and elaborated on quantitative data.It provides vital feedback on areas in which improvement was needed.Specifically, results indicate that students' perceive 7 areas of deficiencies / needs that the program was not currently meeting: grant writing; earlier or second internship experience; earlier awareness of internship requirements; information about careers / jobs after graduation; public health research methods course; summer internship; and continued work on student writing and oral communication.Based on these needs, faculty were able to re-evaluate current curriculum, course offerings, course assignments and outside programming and services to help fill these needs.Specific changes were created and implemented into the Public Health program creating a feedback loop in hopes of closing the gaps between needs and availability (see table 6).This type of assessment highlights the value in a constantly flowing process with the ultimate goal of student learning.
Past research conducted on the role of feedback through the use of e-Portfolios suggest that educators are still in the formative stages of implementation of e-Portfolios and that further research is required to explore its future role, especially regarding feedback in the healthcare arena (Peacock, Murray & Scott, 2011a& 2011b).In support, Perks and Galantino found that the development of a rubric to assess students' e-portfolios in the Holistic Health Minor was a vital mechanism in providing valuable feedback to all faculty teaching in the minor.The rubric also provides an explicit measure of program outcomes in student learning and the use of an e-Portfolio is the ability to share information with multiple faculty readers across disciplines.The current study continues to provide support for the benefits of e-Portfolios in the use of student and program assessment.Specifically, in showcasing students' areas of achievement along with areas they may need improvement.In addition, this type of assessment process provides feedback on program needs and ways in which faculty can implement changes to help address these needs.
In conclusion, this data contributes to the growing body of literature that supports the use of e-Portfolios to enhance student learning and as a measure of program assessment.Although the use of e-portfolios can be time consuming, with some training, adequate time, and specific student criteria, they are certainly worth the investment.The advantages of this technology far outweigh the cost; this study supports this claim in two ways.First, this study provides specific information on the development and implementation of e-Portfolios in a Public Health program.Second, it provides 3 years of quantitative and qualitative data illustrating the use of e-portfolios in assessing students' proficiencies on 13 specific competencies.Students' self-rating along with josotl.indiana.edufaculty's rating on these variables provides descriptive statistics on students learning and on program objectives.Qualitative data supports and builds upon program needs that are not being met.Based on both types of data, program solutions are identified and implemented creating an assessment feedback loop.The use of student e-portfolios is a critical element in this process.
Crowell and Calamidas Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 4, August 2016.64 josotl.indiana.edu Crowell and CalamidasJournal of the Scholarship of Teaching and Learning, Vol. 16, No. 4, August 2016.65 josotl.indiana.eduRQ1: What level of proficiency on Core Public Health Goals and Objectives do students feel they have met upon graduation?RQ2: What level of proficiency on Core Public Health Goals and Objectives do faculty feel students have met upon graduation?MethodologyGiven that assessment is a critical component in the success of an academic program, the development of an initial assessment protocol for the PUBH program was completed during the 2008-2009 academic year.To begin, each faculty member submitted outcome expectations for specific PUBH competencies for each of their classes

Table 1 . Core Competencies A. Students will demonstrate effective oral and written communication skills
 Students will demonstrate effective oral communication skills and effectively utilize electronic methods for communication  Students will demonstrate effective written communication skills B.

Students will have an understanding of the principles of Public Health practice
 Understanding of fundamental techniques relevant to research in Public Health  Perform basic Computational Analysis: portfolio includes examples of student generated analysis of data for internship site projects  An understanding of Epidemiological methodology: Students will be able to accurately describe in the epidemiological principles that are utilized in their internship experiences. Demonstrate an understanding of basic marketing principles: students will demonstrate the ability to apply marketing principles when planning, implementing, and evaluating public health initiatives  Students will have knowledge of Public Health in relation to social, historical, political, and/or economic contexts: demonstrate personal knowledge of the social (i.e. the influence of cultural beliefs on health care practices), historical (i.e.evolution of existing health care structure),political (i.e.influence of governmental policies on access to health care), and economic (i.e.influence of socioeconomic status on the quality and quantity of available health care) dynamics that influence Public Health practice.C.

Students will be able to demonstrate appropriate, ethical and professional behavior when in a role as a Public Health Professional
Students will demonstrate Cultural competence when working in the Public Health Sector: "Cultural competence is a set of congruent behaviors, attitudes, and policies that come together in a system, agency, or among professionals that enables effective work in cross-cultural situations" What is Cultural competence.(2005, October 10).
The use of critical thinking in professional decision making: A proficient critical thinker employs not only logic but broad intellectual criteria such as clarity, credibility, accuracy, precision, relevance, depth, breadth, significance and fairness, when making decisions or during problem solving.Crowell and Calamidas Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 4, August 2016.66 josotl.indiana.edu

Table 2 .
Assessment ScalesUpon completion of students for the Public Health undergraduate degree and based on your internship eportfolio and oral presentation, students are asked to please indicate what they feel is their level of proficiency for each of the following (0=not proficient, 10 = extremely proficient).Faculty are asked to do the same, once they have viewed their e-portfolios.Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 4, August 2016.Feel free to make suggestions/comments about your experience as a Public Health Major (things that you liked, things that you think should be changed, things that should be added, what was beneficial to your learning, and what was not, etc…) Please address how the changes would improve the major.Thank you for your contributions to the major.
Students will demonstrate effective written communication skills #1: Students will demonstrate effective oral and written communication skills Objective: Students will demonstrate effective Oral communication skills and effectively utilize electronic methods for communication Crowell and Calamidas Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 4, August 2016.68 josotl.indiana.eduObjective:

#2: Students will have an understanding of the principles of Public Health practice Objective: understanding
of fundamental principles relevant to current knowledge and research in Public Health Objective: perform basic Computational Analysis Objective: understand Epidemiological methodology Objective: demonstrate an understanding of basic marketing principles Objective: Students will have knowledge of Public Health in relation to social historical, political, and/or economic contextsBoth faculty and students' assessment of the above objectives were comparable, with faculty tending to be slightly lower; averages ranged between the score of 7-8 for all objectives.
Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 4, August 2016.69 josotl.indiana.edu

Students will be able to demonstrate appropriate, ethical and professional behavior when in a role as a Public Health Professional Objective:
Demonstrates appropriate behavior: shows up for work consistently and on-time, follows instructions by carrying out duties, and displays professional behavior and attitude while working as a Public Health Professional Objective: to use critical thinking in professional decision making Objective: Students will demonstrate Cultural competence when working in the Public Health Sector Objective: Students will develop the ability to demonstrate leadership when applicable Crowell and Calamidas Journal of the Scholarship of Teaching and Learning, Vol. 16, No. 4, August 2016.70 josotl.indiana.edu