Review

Programmatic assessment of student experiences using a longitudinal survey

JoLaine Reierson DRAUGALIS, Cecilia M. PLAZA.

ABSTRACT Objectives: The purpose of this study was to gather longitudinal data on a pharmacy class cohort concerning programmatic components as well as develop a profile of student experiences as they progress through the curriculum. Methods: The Class of 2006 (n = 67) completed a 2 page instrument at the conclusion of the first 3 professional years (PY) concerning hours spent in various activities, overall quality of various programmatic components, relationships with others in the college, and employment information. During senior week, a more extensive exit survey was administered. Results: At the conclusion of PY1, 56.5% of the class was working as a pharmacy intern. By PY3 this increased to 88.1% with a decrease to 65.7% in the final year. The hourly range of hours worked followed the same pattern. The rating of Dean's Office Staff and interactions with faculty members remained constant across all 4 years. In the final exit survey the 2 lowest rated program components were the quality of the interaction with assigned faculty advisor and the availability of professional electives. There was no difference across the professional years for the quality of relationships with staff or faculty; however, the mean rating of the quality of relationships with other students was higher for PY4 when compared to PY1. Conclusions: College faculty, administrators, and committees have used the information gathered from students in program assessment and enhancement efforts. Longitudinal data collection allows for tracking of changes and interventions. Key words: Longitudinal Studies. Students, Pharmacy. Education, Pharmacy. United States.

RESUMEN Objetivos: El propósito de este estudio fue recoger datos longitudinales de una cohorte de una clase de farmacia relativa a los componentes programáticos así como desarrollar un perfil de las experiencias de los estudiantes y su progreso durante el currículo. Métodos: La clase de 2006 completó un instrumento de 2 páginas al acabar los 3 primeros años profesionales (AP) relativo a las horas empleada sen varias actividades, calidad general de los diversos componentes programáticos, relación con otros en la facultad, e información sobre empleo. Durante la semana de los mayores, se administró un cuestionario más extenso. Resultados: Al final del AP1, el 56,5% de la clase estaba trabajando en una farmacia como interno. En el AP3 esto se incrementó al 88,1%, con una disminución al 65,7% en el último año. El rango de horas trabajadas siguió el mismo patrón. El ratio de interacciones con la oficina del Decano y con el personal de la facultad permaneció constante durante los 4 años. En la encuesta del final los dos componentes menos valorados fueron la calidad del a interacción con el consejero del a facultad asignado y la disponibilidad del as opciones profesionales. No hubo diferencia durante los años profesionales en cuento a la calidad de la relación con el personal o los profesores; sin embargo la media de puntuación del a calidad del a relación con otros estudiantes fue mayor en el AP4 que en el AP1. Conclusiones: Los docentes del a facultad, los administradores y los comités han utilizado la información recogida de los estudiantes para evaluar el programa y los esfuerzos. La recogida longitudinal de datos permite seguir los cambios y las intervenciones. Palabras clave: Estudios longitudinales. Estudiantes de farmacia. Educación en farmacia. Estados Unidos.

---

JoLaine Reierson DRAUGALIS, PhD. Professor and Dean. The University of Oklahoma College of Pharmacy. Oklahoma City, OK (USA)
Cecilia M. PLAZA, Pharm.D., PhD. Director of Academic Affairs and Assessment, American Association of Colleges of Pharmacy. Alexandria, VA (USA).

 

INTRODUCTION

Echoing Standards 2000, the new Accreditation Council for Pharmacy Education (ACPE) Accreditation Standards in Guideline 22.3 states that student perspectives should be obtained via various methods such as exit surveys and, "…should be systematically analyzed and used to improve all aspects of the program and allow for longitudinal and cross-program evaluation".^1,2 ACPE Standards No. 15 - Assessment and Evaluation of Student Learning and Curricular Effectiveness, Standard No. 21 - Program Information, and Standard No. 22 – Student Representation and Perspectives call for assessment of the student perspective on the harmony of relationships among students, faculty, and staff, availability of financial aid information, and the quality of the Student Affairs Office among other programmatic components. Also little is known about how students spend their time in the "typical" week or the types of external pharmacy work experience gained if any as they progress through the curriculum. The significance of this longitudinal analysis of student perspectives and experiences is important given categories and competency statements in Standards 2007 and the lack of information on how students spend their time and experiences and competencies achieved outside the classroom in venues such as health fairs and work experience as they progress through the curriculum. The literature contains examples of how schools and colleges of pharmacy comply with other components of the ACPE Accreditation Standards but not the elements examined in this study. Using a reasoned approach in assessing student experiences and growth can produce valid data to be used in "guiding policy and decision making, improving practice, and demonstrating effectiveness to external authorities".³ The purpose of this study was to gather longitudinal data on a pharmacy class cohort concerning programmatic components as well as develop a profile of student experiences as they progress through the curriculum.

 

METHODS

The existing anonymous annual student exit survey developed by the then Associate Dean for Assessment and Evaluation at the University of Arizona College of Pharmacy (UACOP) was modified in spring 2003 to track students across the didactic portion of the curriculum. The instrument was administered to students in the graduating class of 2006 at the conclusion of each academic year beginning the spring semester of professional year 1 (PY1). Students present in class the day of the administration were asked to participate. The anonymous database contains information on four areas: 1) number of hours spent weekly in various activities (eg, studying, socializing, caring for family members, and working), 2) quality of various programmatic components (eg, availability of financial aid information, availability of Dean's office staff), 3) rating quality of relationships among students, staff, and faculty, and 4) whether students are currently employed as a pharmacy intern and in what type of practice setting (eg, community, institutional). The adapted exit survey contained core sections administered across all years as follows: 7 items related to the number of hours spent in various activities, 6 items concerning the quality of various programmatic components, 3 items on the quality of relationships among students, staff, and faculty, and 3 items related to pharmacy experience. The survey administered during the first professional year had 3 additional items beyond the core sections on the quality of various programmatic components (eg, pre-pharmacy advising, admissions, and orientation week) unique to that year in the curriculum. The complete exit survey was administered to the class of 2006 which in addition to the core sections contained an additional 8 items addressing the quality of various programmatic components as well as 7 items concerning the administrative, staffing, and resources related to the experiential component and 5 items on the longitudinal drug information to address the portion of the curriculum unique to the final year in the program. As well open-ended questions asked graduates to provide responses on areas needing improvement in the longitudinal program. This information was used as a component of programmatic evaluation at the UACOP. Given the completely anonymous nature of the survey, each administration was considered a cohort that provided a cross-sectional snapshot. A longitudinal analysis was conducted of trends across different items at the cohort level since it is impossible who to know responded on an individual level.

A one-way ANOVA was used to analyze continuous level data across multiple year cohorts with a Tukey or Dunnett post-hoc test for significant F-tests. An independent t-test was used for continuous level data where only 2 cohort years were available. A two-sample test of proportions was used for discrete level data to analyze across cohorts. Data were analyzed using SPSS Version 11.0 (SPSS, Inc., Chicago, IL). The a priori level of significance was set at alpha equal to 0.05. Qualitative data were grouped by theme. The University of Arizona Human Subjects Protection Program declared this project exempt.

 

RESULTS

Subjects

The graduating class of 2006 consisted of 67 individuals, 41 females (61%) and 26 males (39%). Upon admission, 38 (57%) had bachelors degrees and 1 (1.5%) individual had a masters degree. The average matriculant was 24.5 (SD=5.1 years) and ranged from 19 to 49 years of age.

Number of hours spent weekly in various activities

Students were asked to indicate a range for the number of hours spent in the following activities in a typical 7-day week: preparing for class or rotation (eg, studying, reading), working, participating in co-curricular activities (eg, professional organizations, social fraternities), and relaxing and socializing (eg, watching TV, partying, exercising). The hour ranges were: 0, 1-5, 6-10, 11-15, 16-20, 21-25, 26-30, and more than 30 hours. The distribution of the range of hours spent weekly by students in various activities by professional year are shown in Table 1. Except for the number of hours spent preparing for class or rotation and the number of hours spent working, the mode number of hours spent in the remaining activities appeared to be fairly constant across professional years. The mode hour range spent preparing for class or rotation appeared to decrease in the PY3 and PY4 cohorts relative to PY1 and PY2. The mode hour range for working increased in PY2 and PY3 and then decreased again in PY4.

Quality of various programmatic components

The mean student ratings of the overall quality of various programmatic components are shown in Table 2. Five programmatic components were tracked across all 4 professional years. The remaining 9 programmatic components were examined at the most logical points in the curriculum (eg, the Career Day Fair was only offered during PY4 and thus only asked at the conclusion of PY4). The mean ratings of overall quality of the Dean's Office staff and interaction with faculty members remained constant across all 4 years. The mean rating of the overall quality of the availability of (p=0.001) and information on (p=0.0001) financial aid was greater in PY4 than PY1. Students rated the overall quality of the Office of Student Affairs highest in PY4 compared to PY1, PY2, and PY3 (p<0.0001). The mean rating of instruction was lower in PY2 versus PY3 (p=0.035).

In comparing programmatic components only examined for PY4, the 2 lowest rated programmatic components were the overall quality of the interaction with the assigned faculty advisor and the availability of professional electives in the didactic portion of the curriculum (p<0.0001). The overall interaction with faculty members was rated significantly higher than was the interaction with assigned faculty advisor (p<0.0001). The mean rating of elective availability in the final year was higher than the rating of didactic elective choices (p<0.0001). The lack of availability of professional electives in the didactic portion of the curriculum was the second most frequently mentioned aspect of the pharmacy program that they would want to improve.

Rating quality of relationships among students, staff, and family

The rating of the quality of relationships among students, staff, and faculty, is shown in Table 3. There was no difference across professional years for the student ratings of the quality of relationships with staff or with faculty. The mean rating of the quality of relationships with other students was higher for PY4 compared to PY1 (p=0.047). These ratings appear to be consistent with the qualitative comments provided by students about what they liked the most about their pharmacy education with faculty ranking the highest in number of mentions and friendships/classmates ranking fourth.

Student employment status and practice setting information

The descriptive statistics for the proportion of students working in a pharmacy practice setting for each professional year are shown in Table 4. A little more than half the class (56.5%) worked in a pharmacy setting during PY1. The proportion of students working increased in PY2 (p=0.030) and remained steady in PY3. In PY4 the proportion of students working decreased compared to PY2 and PY3 (p=0.0005). The greatest proportion of students worked in the chain/retail community pharmacy setting across all 4 professional years. Based on information collected by the Student Affairs Office for those that were able to indicate their plans after graduation approximately 50% entered chain/retail community pharmacy practice as their first job upon licensure.

Additional Qualitative Feedback from Students

In final data collection, students were asked to list the 3 things they liked the most about their pharmacy education. The most frequently mentioned aspects of their pharmacy education students liked the most listed in descending order were: the faculty, experiential rotations, the curriculum as a whole, friends/classmates, and student organizations/extracurricular activities. Students were also asked to list the 3 things that they liked the least about their pharmacy education. The most frequently mentioned aspects of their pharmacy education students liked least listed in descending order were: the longitudinal drug information rotation, the physical facilities (eg, small classrooms, uncomfortable seating), student reflective portfolios, and the senior capstone project. The final open-ended item asked students to list the 3 things they would do to improve their pharmacy education. The most frequently mentioned aspects students listed as areas they would improve in descending order were: rotations offered earlier in the curriculum, improved availability of electives, and the provision of formal preparation for the Board licensure examinations.

 

DISCUSSION

The change in the shape of the distribution and mode number of hours working for pay off campus decreasing in PY4 was expected due to the increased time demands and required attendance during the experiential portion of the curriculum. This was confirmed by a drop in the number of students indicating current employment status during PY4. Of those students that did work in a pharmacy setting during pharmacy school, the majority did so in a community setting. Given the large number of community pharmacies in the Tucson and Phoenix areas, this was not unexpected. By PY3, 89% of students indicated that they were working in a pharmacy setting which could be a potential double-edged sword which in the past has been a concern of the faculty. While having the vast majority of students working by PY3 could provide further reinforcement of material and provide opportunities to complete outcomes based competency statements it has to be balanced with appropriate priority setting on the part of the student such as not working so many hours that it detracts from coursework or affects attendance.

The apparent decrease in the mode number of hours spent preparing for class/rotations dropping in PY3 and PY4 compared to PY1 and PY2 was unexpected given the increased course load particularly in PY4 with a longitudinal drug information rotation running concurrently with other experiential rotations, a senior capstone project requires that a proposal process that begins during PY3, in addition to the annual requirement of the submission of a student reflective portfolio. Numerous responses from students to the open-ended items in listing the longitudinal drug information rotation, senior capstone project, and student reflective portfolio as the most disliked aspects of the pharmacy program indicated that they took too much time. The qualitative statements do not appear to agree with the overall ratings of how as a group they spent their time in PY4, spending apparently less in preparation for class/rotations than in PY1 and PY2. The item may not have reflected actual time spent for rotation preparation given that some of the time spent preparing may have been incorporated into the rotation time. The qualitative item on the instrument may have indicated a potential problem with the way subjects were being asked to quantify their preparation time.

The higher rating of the Student Affairs Office in PY4 may be the result of the Student Affairs Office serving as one of the main points of contact while students are on rotation, helping to coordinate ensuring students have the required paperwork required for licensure. The Student Affairs Office also serves as the focal point for planning graduation festivities and the actual administration of the exit survey is conducted during "senior week" where all PY4 students are together one final time for a week prior to graduation. It is possible there was also somewhat of a halo type effect given the timing of the administration. The increased rating of the availability of and information on financial aid in PY4 relative to PY1 could have resulted from an increased number of named scholarships available to students in their final year, the increased dollar amount of awards as students progress further in the curriculum, changes instituted simplifying application procedures, and/or increased skill at navigating the financial aid system by the final year.

The low rating of interaction with assigned faculty advisor is a continuing concern. In spring 2006, faculty advisors assumed the responsibility of grading their advisees' student portfolios. It is hoped this increased and sustained interaction will result in a higher rating of this program component. The finding that graduates felt that there were insufficient professional electives has also been encountered in previous years' exit surveys. Two new didactic electives were developed and offered in academic year 2005-2006. Faculty must have the time, desire, and incentives to provide elective offerings.

Of the qualitative feedback from students, wanting earlier rotations, the provision of formal preparation for the Board licensure exams, and the need for improved physical facilities had not made the top 5 in previous graduating classes. The physical facilities could be an artefact of the transition to the new UACOP building that occurred while the class of 2006 was on rotations which solved many of the concerns they listed such as cramped classrooms and certain seating areas in poor condition. The lack of formal introductory pharmacy practice experiences should be addressed with necessary curricular reform through more explicit requirements in Standards 2007 as to what constitutes introductory practice experiences. The listing of the provision of more formal preparation for the licensure examinations was unexpected and seemingly unfounded as outcomes were no different from previous years.

 

CONCLUSION

This longitudinal study provided valuable information on student perceptions of various programmatic components as well as in how students spend their time. College faculty and administration have been given summaries of the findings to be used in the next accreditation self-study deliberations and on-going curricular evaluation. The College's Curriculum and Evaluation and Special Studies Committees have used relevant sections and previous exit survey results in their on-going deliberations.

 

DISCLAIMER

The views expressed in this paper are those of the authors and do not represent those of the American Association of Colleges of Pharmacy.

 

CONFLICT OF INTEREST

None declared.

 

References

1. American Council on Pharmaceutical Education. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree. Chicago, IL: The American Council on Pharmaceutical Education Inc; 1997.         [ Links ]

2. Accreditation Council for Pharmacy Education. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree. The Accreditation Council for Pharmacy Education Inc. Available at: http://www.ACPE_Revised_PharmD_Standards_Adopted_Jan152006.pdf (accessed on March 10, 2006).         [ Links ]

3. Kuh GD, Gonyea RM, Rodriquez DP. The scholarly assessment of student development. In: Banta TW, ed. Building a Scholarship of Assessment. San Francisco: Jossey-Bass; 2002: 100-127.         [ Links ]