INTRODUCTION

The first Teaching and Learning Curriculum, also known as a Residency Teaching Certificate Program (RTCP), for pharmacists was developed at the University of Kentucky in 2001.¹,² RTCPs were developed to educate residents on academia related content and provide various teaching experiences, providing a structured approach to training future educators. Almost two decades later, RTCPs have grown in number throughout the United States. A study completed in 2019 by Islam et al., found that out of 132 surveyed colleges of pharmacy, RTCPs for pharmacy residents were offered by 69 (52%).³ There is currently no requirement for individual RTCP accreditation from a governing body, though there have been several guidance documents and suggested guidelines published to direct programs from the American College of Clinical Pharmacy (ACCP), the American Association of Colleges of Pharmacy (AACP), and the American Society of Health-System Pharmacists (ASHP).⁴-⁶ In the authors' experience, the plans, topics, and activities are often “handed-down” from past coordinators to new coordinators without much quality improvement planning and the specific curriculum of each RTCP nationwide varies based on interpretation of the guidance documents.⁵

Some colleges of pharmacy within an individual state have collaborated to provide a state-level program which subsequently provides standardization amongst the state at a local level. Certain states, such as Indiana, have tried this united approach and have adopted a statewide standardization for their RTCPs. This consistency allowed the Indiana Pharmacy Teaching Certificate Program (IPTeC) to provide quality instruction, mentorship, and opportunities for presentations and precepting.² That being said, many states offer regional RTCPs that are each associated with separate colleges of pharmacy within the state.

Ohio Northern University is one of seven Accreditation Council for Pharmacy Education (ACPE) accredited colleges of pharmacy in Ohio and each of the seven colleges coordinates its own RTCP.⁷ A prior survey of Ohio RTCP leaders has shown there is variation in the content included and in how each of those programs are delivered.⁸ The RTCP at Ohio Northern University was pioneered in the fall of 2009 and the specific requirements for completion of the program have evolved over the past decade. Currently, the program is standardized such that all residents have the same minimum requirements for completion which align with the three pillars of teaching, service, and scholarship. This consists of in-person seminar days, online supplemental material, hands-on activities, and self-reflection with the majority focused on didactic teaching. Although it has varied in the past, in the 2021-2022 academic year, the RTCP has participants from eleven different residency programs. The evolution of the program has resulted from informal feedback from residents, residency directors, teaching mentors, and program coordinators. However, there has not been a formal programmatic assessment over the 10 years of its existence. The authors set out to determine what activities and seminar topics were found beneficial by past Ohio Northern University RTCP participants, both during and after completion of the program.

The primary objective of this quality improvement project was threefold: 1) to identify past RTCP participants' perceptions regarding the program seminars, activities, and requirements; 2) to determine the short-term and long-term impact on their careers and interaction with learners; and 3) to improve the program to meet participants' needs.

METHODS

This survey, granted exempt review status through Ohio Northern University's IRB, was developed by the authors and was administered through Qualtrics. There were 25 items with 6 overarching themes. The sections were named as such: 1) consent and demographic info; 2) overall improvement; 3) mentors; 4) opinions while in the RTCP; 5) opinions based on current role, and 6) open-ended final questions. The types of questions that were used included multiple-choice, fill-in-the-blank, select all that apply, Likert scale, sliding scale, and matrix questions where participants were given the option of answering beneficial, not beneficial and not applicable to various RTCP components. Participants were not required to answer all questions of the survey. Prior to officially releasing the survey to RTCP participants (September 16, 2019), it was reviewed for grammatical revision and question input by three Ohio Northern University college of pharmacy faculty that had not previously participated in Ohio Northern University's RTCP to gain objective insight on the survey's question design. The researchers asked the pilot participants to ensure subject matter, readability, and length were appropriate for the survey. Minor changes were made to the survey as a result of the pilot test feedback to provide clarity and improve the organization.

Once the survey was ready to disperse, the survey was sent to all participants that completed Ohio Northern University's RTCP from the inaugural 2009-2010 class to the 2018-2019 class. Past participant email addresses were obtained and the survey was sent as an email directly from the Qualtrics System in late September 2019. Participants were given 12 days to complete the survey with two reminder emails sent 8 and 11 days after the initial email through the Qualtrics system. After 12 days (September 30-October 11, 2019), the survey was closed to further responses. The surveys were completed anonymously and there was no compensation for participation. Data was compiled and sorted by question. Researchers then analyzed and coded the responses to each question and presented them in aggregate form.

RESULTS

A total of 93 residents have completed the RTCP at Ohio Northern University. Eighty-nine emails successfully reached former Ohio Northern University RTCP participants (3 emails bounced back, 1 email address unavailable) and a total of 41 survey responses were received for a 46.1% response rate. Prior to participation in the RTCP, participants attended ten different pharmacy schools across three different states with the majority of respondents having attended Ohio Northern University (58.5%). Pharmacy school graduation years varied from 2009-2018 and similarly, RTCP graduation years varied from 2010-2019. When asked if the ability to participate in a RTCP affected the residency site pursued, 30 respondents (73.2%) indicated that it did. The amount of time elapsed since the completion of the RTCP was evenly distributed with 21 respondents (51.2%) completing the RTCP within the last 5 years and 20 respondents completing the RTCP between 5 and 10 years ago (48.8%). All participants completed a PGY1 residency with representation from eleven different PGY1 residency sites. Nine respondents (22%) completed a PGY2 of various specialties representing seven different states.

Information regarding participants' current career positions and length in position can be found in Table 1. Of note, some respondents included more than one current position, however if an academic title was given, then they were included with the academic cohort only. If multiple positions were given with different lengths of time by a participant, the authors tried to choose first, the length of time that correlated to the current practice site if known. If unknown, authors chose the longest length of time of the positions provided. Some examples of non-academia roles provided by respondents included: critical care clinical pharmacist, ambulatory care clinical pharmacists, lead pharmacist, oncology clinical pharmacist, pharmacy manager, etc. Regarding their current position, data about participants' roles with learners (Table 1) and percentage of time each week spent with learners was obtained. In regards to role with learners, a total of 128 responses were provided by 41 participants as it was a “select all that apply” question format. Highest involvement is around precepting APPEs (82.9%) and PGY-1 residents (56.1%). Time spent with learners varied widely, ranging from 10 - 100% of their time each week, with an average of just under 50% (46.6%) of their time.

Participants were also asked questions addressing changes from “prior to” to “after” completing the RTCP. Respondents used a 5 point Likert scale, ranging from 1 (very unlikely) to 5 (very likely), to answer the question “How likely were you to pursue a career in academia?” both prior to (N=38) and after completing (N=40) the RTCP. The median of the responses increased from 3 to 4 when comparing “prior to starting” and “after completing”, respectively. Participants were also asked to select via a 5 point Likert scale, ranging from 1 (completely disagree) to 5 (completely agree), their agreement with the statement “I felt prepared and confident to educate others after the completion of the RTCP” both prior to (N=39) and after completing (N=41). The median of the responses increased from 2 to 4 when comparing “prior to starting” and “after completing”, respectively. When asked about whether or not all residents should complete a residency teaching certificate program, 56.1% (23/41) agreed that it should be a requirement. When further asked if all RTCPs should be standardized throughout the country, meaning that all programs would have the same structure, information presented, and activities completed 61% (25/41) agreed that RTCPs should be standardized. Reasons that respondents cited for not recommending RTCP standardization included the potential for programs to have decreased flexibility for residents, decreased creativity, quality, variety and passion, and unequal resources.

Since there would potentially be a difference between the benefit of certain aspects of the program while in the program versus while in their current role, the participants were asked to reflect back on individual activities and seminar topics and identify if they were beneficial or not beneficial while a resident in the program and then again in regards to their current career. Since the program has evolved slightly over time, the participants were also able to indicate “not completed” if they personally didn't complete that component. The results for those participants that had completed the seminar or activity are summarized in Table 2 and Table 3 in regards to being beneficial or not beneficial. Of note, no participants found the seminar topic “Legal Issues with Academia (FERPA, Title IX)” beneficial in their current roles after completing the RTCP despite many respondents indicating that they currently work with all learners (IPPEs and APPEs). Participants (n=35) used a sliding scale to identify what percentage of the overall RTCP should be spent discussing seminar topics compared to hands-on activities and the average response was 42.3% (range 20% - 80%). When asked through a free text response if there were any other activities, not previously mentioned, of the RTCP that were beneficial to their current position, 16 respondents provided feedback, a majority of which stated there were no additional activities (n=14). Participants were asked to provide any free text suggestions to improve future design of RTCPs. A total of 18 participants responded, some including more than one suggestion. Suggestions have been summarized in Table 4.

When respondents were asked about their assigned RTCP teaching mentor, fourteen different mentor names were provided by participants highlighting a variety of viewpoints because there was interaction with many different mentors. When asked about their current interaction with their mentor, 57.1% (16/28) said they were no longer in contact with their RTCP teaching mentor. As a free response question, respondents were asked to reflect on the beneficial aspects of having a teaching mentor. Responses were classified by the authors as teaching advice/ feedback (T), career info (C), encouragement/ empowerment (E), or not helpful (NH). Twenty eight responses were coded by the authors and responses could be classified as multiple types leading to a total of 36 coded categorizations. Overwhelmingly, the most beneficial aspect of having a teaching mentor was the teaching advice and feedback provided by their mentor. One respondent stated having a teaching mentor was not beneficial. Respondents were also asked to identify ways in which their teaching mentor may have provided better assistance or improved their experience in the RTCP. Eighteen respondents did not have any suggestions while five respondents suggested increased communication or more one-on-one meetings with their mentor. In general, two respondents requested the ability to partake in more academia activities.

DISCUSSION

There are a variety of factors to consider when interpreting and drawing conclusions from the results of the survey. Overall, the response rate was relatively high and included at least one participant from each year the program was offered, providing a variety of perspectives from prior RTCP participants. The survey respondents had completed the program between 2010 and 2019, providing feedback from a decade worth of program delivery. As with most programs there has been a natural shift and evolution in the program over time in terms of activities and seminars included, content delivery methods, and faculty involved. Although this extended timeframe could lead to potential recall bias, it is hoped that the spread of responses from different RTCP years will help to minimize the impact of this bias.

A majority of survey respondents indicated the option to complete an RTCP during residency affected the residency program pursued. When asked directly to rate their interest in pursuing a career in academia prior to completing the RTCP, the median was 3 representing “neither likely nor unlikely”. Interestingly, when asked about their interest in pursuing a career in academia after completion of the RTCP, there was an increase in the median response to 4 (“somewhat likely”). This demonstrates a growth in participant interest in this career path, which is one of the program's overarching goals. Another beneficial aspect of the program is that participants did see an increase in how prepared and confident they felt in educating others after completing the program, compared to before, with the median score increasing from 2 (“somewhat disagree”) to 4 (“somewhat agree”). The perceived increase in confidence will be relevant in all career paths of pharmacy, as providing education is not limited to students in the classroom but can include educating healthcare professionals, patients, and the public. All respondents identified at least one way that they interact with learners in their current role. Although many participants do not have a full time position in academia, they are still working with learners in a variety of ways throughout their work week. This demonstrates that RTCPs can be beneficial to pharmacists, even if they are not practicing in an academic setting specifically but work with learners in other areas such as pharmacy preceptors or pharmacy industry fellows.

Some of the more traditional academia activities and seminar topics (developing test questions, meeting with the teaching mentor, effective assessment and feedback) were reported to be beneficial during the program. When reflecting on these same traditional academia activities in their current roles, residents did not identify these as top activities that were helpful. Some activities and seminar topics (assessing students on an assessment day, participating in educational journal clubs, delivering lunch and learn presentations, reflection) that did not seem beneficial to the residents during RTCP completion were viewed to be helpful to residents' current roles. Some activities and seminar topics (delivering professional CE, giving a large room lecture, and facilitating small group learning) stood out as most beneficial both during and after the RTCP. There were also activities and seminar topics that were not beneficial both during and after RTCP completion such as admissions and marketing and institutional and programmatic finance. The research team does not encourage program coordinators to use these results solely in determining which topics should be included or removed but rather, should use the information obtained to re-evaluate program content and distribution of time allocated to the different topics. For example, the least beneficial topics could be delivered in a different way (i.e. case-based scenarios) that apply to settings outside of the traditional classroom and reflect other career paths such as an experiential preceptor or residency director. Furthermore, RTCP coordinators should help residents identify why the covered topics are important, even if residents do not end up in academia. For example, less than 30% of respondents from this survey have a current position in academia, however all participants still spend a large quantity of time with learners. Finally, this data should be a reminder to programs to self-reflect on the purpose of each topic included and how it contributes to the program's overall mission for participants' both short-term and long-term growth. Based on the specific goal of the RTCP, coordinators should consider whether their program is intended to be standardized for all participants or specific to participants' goals. For example, coordinators could consider removing some of the specific academia content as a requirement for all participants and create a “track” type program for those who are interested in an academia career.

Respondents were asked more questions specifically about their assigned RTCP mentor with various experiences and viewpoints expressed. Although over half of the respondents said they were no longer in contact with their mentor, meeting with the teaching mentor was in the top five most beneficial activities during the RTCP. Overwhelmingly respondents found their mentor beneficial. Some suggestions for improvement included: increased communication, more one-on-one meetings with their mentor, and the ability to partake in more academia activities. Currently, participants are paired with mentors based upon their large room lecture topic assignment and faculty availability. It would be interesting to see if the mentor-mentee relationship would be further improved by pairing compatible partners through established personality tests, providing training to the mentors with clear expectations and minimum standards, and scheduling quarterly one-on-one in person meetings when residents are on campus for the seminar days.

After completing this quality improvement project the researchers plan to take the data to adjust the program and better meet the needs of today's participants. Specific quality improvement ideas that have come from this work include redistribution of time in seminars compared to hands-on activities, the adoption of tracks or concentrations within the RTCP, and the creation of mentor training and development. Further research and quality improvement into the needs of residency programs and academia could better round out the landscape that RTCPs exist amongst. While not all of the participants ended up in academia they all expressed having some interaction with learners and overwhelmingly found the program useful, which is encouraging that the program is at baseline meeting their needs. Researchers plan to continue quality improvement and also adapt and adjust the program based on the results to further increase the usefulness of the program.

CONCLUSIONS

Through this survey, the perceptions and beliefs of just under half of resident participants of a single RTCP were obtained regarding how beneficial the programming, activities, and mentorship offered through the program were both during and after its completion. Future research ideas that have emerged during this project are two-fold. First, identifying the desire and creation of a standardized minimum competency for assessment of RTCP participants and RTCPs holistically. Second, since our results showed benefit to those not in traditional academia roles, it begs the question if RTCPs should be expanded to other healthcare educators and non-resident pharmacists to develop more well-rounded educators.