Original Research

Identification by observation of clinical pharmacists' activities in a hospital inpatient setting

Peter STUCHBERY, David CM. KONG, Giovanna N. DESANTIS, Sung K. LO.

ABSTRACT The aim of the study was to quantify clinical pharmacists' workload in Australia. Specific objectives were to perform a direct observation of the pattern of clinical activities in two acute hospitals and compare that with previously documented self-reported patterns. We were also interested in identifying what records kept by pharmacists would capture all the activities they perform. Methods: An observer recorded the activities of clinical pharmacists on six separate days in the medical and surgical wards of two Melbourne metropolitan hospitals. We examined resultant data to determine suitable records by which clinical pharmacists could capture all the activities they perform. To compare the observed pattern of clinical activities with those earlier self-reported by pharmacists, we categorised our data using the Pharmacy Activity Codes present in the penultimate version of the ICD-10-AM classification system. Results: The observer recorded the performance of 807 workload 'events', representing 28 separate types of activities. When compressed into the Pharmacy Activity Codes formerly used in the ICD-10-AM classification system, the pattern of activities identified by direct observation matched that which had previously been self-reported by pharmacists. The majority of the activities performed could be captured by completion of a Pharmaceutical Care Plan and by recording pharmacists' interventions to a database. The remaining activities may be recorded for departmental workload purposes in a simple template format. Conclusion: The pattern of clinical pharmacist activity at the two hospitals was confirmed by direct observation as similar to that previously reported in other Australian hospitals. A Pharmaceutical Care Plan, a database for intervention recording and a simple workload template provide the means to record all activities that clinical pharmacists perform. Key words: Workload. Hospitals. Pharmacists. Australia.

RESUMEN El objetivo del estudio fue cuantificar la carga de trabajo de los farmacéuticos clínicos en Australia. Los objetivos específicos fueron realizar una observación directa del os modelos de actividades clínicas en dos hospitales de agudos y compararlas con los modelos que habían sido previamente auto-reportados. También estábamos interesados en identificar que registros llevados los farmacéuticos podrían capturar todas sus actividades. Métodos: Un observador registraba las actividades del os farmacéuticos clínicos en seis días separados en los servicios de médicos y quirúrgicos de dos hospitales metropolitanos de Melbourne. Examinamos los datos resultantes para determinar registros adecuados con los que los farmacéuticos clínicos podrían registrar todas las actividades que realizaban. Para comparar los modelos de actividades clínicas con las auto-reportadas anteriormente, categorizamos los datos utilizando los Códigos de Actividad Farmacéutica (CAF) presentes en la penúltima versión del sistema de clasificación CIE-10-AM. Resultados: El observador registró las actividades de 807 eventos de trabajo, que representaron 28 tipos diferentes de actividades. Al comprimirlos en los CAF anteriormente usados en el sistema CIE-10-AM, los modelos de actividad identificados por observación directa se ajustaban al os que habían sido previamente auto-comunicados por los farmacéuticos. La mayoría del as actividades realizadas podía ser capturada en la cumplimentación de un plan de atención farmacéutica y en registrar las intervenciones del farmacéutico en una base de datos. El resto del as actividades puede registrarse para análisis de carga de trabajo del departamento en un modelo de formato simple. Conclusión: El modelo de la actividad de un farmacéutico clínico se confirmó por observación directa como similar a los previamente comunicado en otros hospitales australianos. Un plan de atención farmacéutica, una base de datos para registro de intervenciones y una simple hoja de registro de carga de trabajo proporcionan los medios de registro de todas las actividades que realizan los farmacéuticos clínicos. Palabras clave: Carga de trabajo. Hospitales. Farmacéuticos. Australia.

---

Peter STUCHBERY. Pharmacy Department, The Northern Hospital, Epping, Victoria, Australia.
David C.M. KONG. Department of Pharmacy Practice, Victorian College of Pharmacy, Monash University Parkville, Victoria, Australia.
Giovanna N. DESANTIS. Pharmacy Department, Western Hospital Footscray, Footscray, Victoria, Australia.
Sung K. LO. Faculty of Health & Behavioural Sciences, Deakin University, Burwood Victoria, Australia.

 

INTRODUCTION

Pharmacists practicing at ward level perform a constellation of tasks – clinical pharmacy services – for hospital inpatients in order to promote the safe and effective use of medicines. The range of services in different practice settings is well described.^1-3 Less well documented is the range of activities that comprise these services and their relationship to individual patient need.

In Australian reports describing how pharmacists document their services, the source of data about the activities of clinical pharmacists is usually derived from records kept by the pharmacists themselves. These data can be recorded by pharmacists in the normal course of their work⁴ or, for specifically determining this information, by use of a self-reporting time-log instrument.⁵ A few reports exist describing the recording by direct observation of the activities of clinical pharmacists, but these reports are derived from elsewhere.^6,7 To our knowledge, there have been no Australian reports concerning the use of an observer to record clinical pharmacists' activities.

Within the Australian practice setting, reports show attempts at documentation for the purpose of describing the input required by pharmacists to provide clinical services. Early reports from Victoria⁸ and Western Australia (WA)⁵ describe the activities performed by clinical pharmacists and include some analyses of the average cost or time involved to provide these pharmacy services per occupied bed. Recently, more reports of clinical pharmacy documentation have appeared at the local and at the national level. Two hospitals in WA reported a method of recording pharmacists' activities that could be quantified by pharmacist, by ward or by medical team.⁴ A national survey of clinical pharmacy documentation practices showed the extent to which pharmacists documented their clinical services.⁹ The authors noted the lack of uniformity between sites in the nature of documentation and found that most documentation quantified only the number of activities performed per ward or per pharmacist. While reporting the number of activities performed for each individual patient took place about one-third of the time, the actual cost (or time) of pharmacy service provision per patient could be reported by only 2.5% of sites.⁹

As a first step in our aim to evaluate systematically pharmacists' clinical activities and their contribution to patient care, we sought to identify the range of clinical activities performed by pharmacists. Our objective was to use an observer to confirm patterns of clinical pharmacy activities that have been self-reported by pharmacists themselves in earlier studies and to identify what records kept by pharmacists of their clinical activities would be needed to capture all of the activities they performed and so quantify the workload of pharmacists as they provide clinical services to individual patients. Australian Health Ministers have recently mandated a process of pharmaceutical review for patients in Australian hospitals.¹⁰ Quantifying clinical pharmacists' workload will thereby be important in establishing the extra resources required to provide this service for all patients.

 

METHODS

We conducted the study in the general medical and surgical wards of two acute general hospitals in Melbourne, The Northern Hospital (TNH) and Western Hospital Footscray (WHF). No specialist wards, such as renal wards, were included. This setting was chosen as it would provide a diverse range of medical and surgical presentations that require many clinical services that pharmacists can provide. In common with metropolitan teaching hospitals in Australia, TNH and WHF provide clinical pharmacy services. These services include the review of medication orders, provision of information on drugs and medicines to both patients and to other clinical staff and the intervention in and monitoring of drug therapy through the identification of potential or actual medication-related problems.³

We identified the actual activities performed by pharmacists by using an observer (in a modified work sampling technique) to record the activities that the pharmacists performed. The observer was a pharmacist with experience in providing clinical services to hospital inpatients and who had previous experience in pharmacy practice research. While we originally sought ethics approval for this study, neither the Human Research and Ethics Committees of Northern Health (TNH) nor of Melbourne Health (on behalf of WHF) required approval for the study in regard to the patients or the pharmacists, as they regarded it as a 'staff audit with minimum patient impact'.

To aid the observer in recording activities without the need for continual handwriting, we compiled a list of 27 types of activities and assembled them on a customized form – the data collection sheet. The activities listed on the sheet were compiled based on literature review and the experience of the researchers. Observed activities that had not been included in the sheet were recorded by way of freehand note on the back of the form. The observer used a separate form to record the activities performed by pharmacists for each patient by circling the "Y" symbols to indicate those events observed. We did not attempt to quantify all (i.e. repeated) activities performed for each patient, nor did the observer record the times taken to perform activities as we sought only to identify the complete range of activities carried out.

Each type of activity observed (including multiple observations of the same activity) was recorded as an 'event'. Observations took place in the medical and surgical wards on three non-sequential days on each hospital site during March and April, 2004. Each day, on each site, the observer accompanied a different pharmacist while they undertook their clinical duties in their allocated wards.

We analysed the data collected by categorizing them according to four major activity groupings associated with different stages of inpatient care - medication review, monitoring activities, provision of counselling or drug information and discharge planning. These groupings were devised to aid in subsequent presentation of the results and showed activities performed by pharmacists at different stages of a patient's stay in hospital. We considered that this may aid in our objective to identify suitable records made by pharmacists that could quantify the activities they performed on behalf of each patient. To enable comparison with previously published work,¹¹ we also categorized data according to the Pharmacy Activity Codes used in the penultimate version of the Australian edition of the International Classification of Disease classification system (ICD-10-AM).¹² The ICD-10-AM is an Australian modification of the ICD-10 coding system developed by WHO. The penultimate version of ICD-10-AM, extant at the time of the earlier study,¹¹ included a number of 'pharmacy activity codes' by which it was possible to record clinical activities by pharmacists under the ICD system.

 

RESULTS

During the study the observer recorded the performance by six pharmacists (three on each site) of 807 workload 'events', representing 28 separate types of activities. These took place on behalf of 195 inpatients; each pharmacist thereby provided a clinical service to an average of 33 patients on each day of observation. Pharmacists performed an average of 4 activities for each patient (range 0-10) on the day of observation. Of these 807 workload 'events', 776 (96.2%) were recorded from the list of activities provided on the data collection sheet. A further 31 events (3.8%) were not accounted for in the list. The most frequently performed activities were review of the medication chart, annotation of the medication chart and review of the patient history, these activities comprising 171 (21.2%), 97 (12%) and 75 (9.3%) respectively of the total events observed (Table 1).

Approximately equal numbers of the 31 events recorded by freehand note took place on each site. On both sites the majority of these events concerned liaison, either between staff or externally with carers or primary treatment providers. The remainder represented stock management and administrative issues.

Examining activities that represented direct contact with the patient or their carer, the data show that pharmacists conducted interviews with the patient or their carer on 36 occasions - or 18.5% of the 195 patients whose treatment was reviewed by a pharmacist. Other activities that represented direct patient contact (provision of drug information, medication counselling and liaison about what medication was required to be dispensed on discharge) occurred on a further 46 occasions (23.6%). Some patients (or their carer) received direct contact from the pharmacist for more than one activity (e.g. medication history interview as well as provision of medication counselling). The total number of patients or their carers with whom the pharmacist made direct contact for one or more reasons was 59 (30.2%). At each hospital, the pharmacist liaised with the patient or their carer for more than one reason (i.e. to seek or provide different information) to the same extent - on 43.5% of occasions at TNH and on 44.4% at WHF.

Table 2 shows the reclassification of the observed activities according to the Pharmacy Activity Codes in the penultimate version the ICD-10-AM disease classification system. This reclassification required the removal of 34 events that represented the pharmacist's role in initiating a change to treatment. These pharmacy 'interventions' were not included in the ICD-10-AM classification system and this amendment reduced the total number of events to 773. Under this reclassification, the largest contributions to observed events came from activities associated with medication chart review (53.7%) and from clinical review activities (21.6%). Each of the remaining seven activity classifications represented less than 10% of observed events, with the largest of these from the provision of drug information (48 events (6.6%)).

There were no significant differences between the frequency of activities performed at TNH and WHF (chi-sq=15.1, df=8, p=0.06; detailed data not shown). Medication order review was the most frequently observed activity at TNH and WHF (51.4% vs. 55.4% of activities respectively) with clinical review activities respectively contributing 26.1% and 18.2% of activities conducted. All other seven activity classifications contributed less than 10% of activities observed on each site, with provision of drug information being the next highest at both TNH (5.2%) and WHF (7.0%).

Table 2 also shows a comparison with pharmacists' clinical activities previously self-reported by pharmacists in a national study.¹¹ Medication order review represented 53.7% of pharmacists' workload in this study vs. 53.1% in the earlier study (p=0.738), with clinical review activities comprising a further 21.6% (this study) and 20.8% (p=0.584) in the earlier study respectively. All other workload activities by ICD-10-AM Pharmacy Activity Codes comprised less than 10% of pharmacists' workload in both studies.

 

DISCUSSION

We believe this is the first Australian study that has used an observer to identify the activities of clinical pharmacists. In capturing activities performed, but not the time involved, we thereby employed a more objective approach and robust sampling technique. Work sampling has been previously described for work measurement in pharmacy research outside Australia.^13,14 In this study, the range of activities performed by clinical pharmacists was in accordance with that expected. We thereby confirmed, by using an observer, the range of activities that clinical pharmacists record themselves using self-reporting records in the Australian setting.¹¹

When designing the data collection sheet, we included as many activities as we could anticipate on this document, so as to facilitate the recording of as many activities as possible. In a previous study, an observer recorded the activities of clinical pharmacists according to a list defined from the authors' experience.⁷ All observations fell within the list although two categories of 'other duties' (e.g. answering pagers for non-ward-related requests) and 'unproductive time' (e.g. time spent socialising with other staff) comprised 10% of recorded activities and thereby represented a potential taxonomy for unanticipated activities. During our study, more than 96% of workload 'events' were recorded by annotating recognized entries on the data collection sheet; less than four per cent of the observed workload represented activities that had not been anticipated. The workload pattern of the pharmacists thereby followed a conventional model of clinical pharmacy practice.¹⁵ In seeking to identify pharmacists' clinical activities and the records by which they could be recorded, this finding provided reassurance that our methodology identified the pattern of pharmacist's activities that typifies practice in a hospital inpatient setting.

The majority of workload events (nearly 70%) were associated with activities concerning the initial review of patients and of the medication ordered for them. This was similar to results reported in other studies.^7,11,16 The relatively high proportion of activities associated with patient discharge is consistent with other reports that have described pharmacist involvement at the time of discharge prescription preparation to promote the optimum and efficient supply of discharge medication.^17,18 We believe that the recent introduction of the Pharmaceutical Benefits Scheme (PBS) for discharge prescriptions in some Australian States (including Victoria) is responsible for this finding in this study. The PBS is a subsidised medicines scheme for patients in the community and for those being discharged from hospital. The technical requirements of prescription writing for the PBS are difficult for junior prescribers and extra pharmacist input into this point in the discharge process has become more common as it facilitates patient discharge.¹⁹

While the majority of workload events performed by the pharmacists concerned the review of medication ordered for patients in the context of the patients' clinical and social circumstances, pharmacists made direct contact with the patient in less than one-third of patients. The study was carried out as a series of 'snapshot' observations on non-consecutive days and, for patients with whom direct contact was not observed, it is possible that patient contact occurred on another day during each patient's admission. However, given the consistency of this observation on six separate days in the high-turnover medical and surgical wards of the two hospitals, it is possible that patient contact was not a principal focus for the pharmacists. If this was the case, it represents a missed opportunity for pharmacists to add value to the process of medication review through their specific knowledge of medications and medication-related problems.^20,21

Reclassification of the observed workload activities according to the Pharmacy Activity Codes used in the ICD-10-AM disease classification system provided confirmation that the practice patterns of the pharmacists in this study did not differ from those observed in a previous national study.¹¹ This previous study sought to confirm that the activities of clinical pharmacists could be reported using the 'pharmacy intervention codes' extant in the penultimate version of the ICD-10-AM classification system.¹² A subsequent revision of this classification system collapsed the existing nine pharmacy activity codes to a single code, rendering the new code structure unsuitable for standardising pharmacy activity reporting. We found no significant differences in the practice patterns of pharmacists at TNH and WHF. In addition, there was close similarity (Table 2) between the self-reported activity patterns of pharmacists in other Australian States and Territories and the patterns recorded by an observer in our study; all these provided additional assurance that our findings are representative of Australian practice and have the potential to be applied in other settings.

Three ICD-10-AM activity codes viz. (i) medication history interview, (ii) patient medication counselling and (iii) advocacy/community liaison, represent a valuable source of information for other current and subsequent treatment providers and thereby should form part of the patient's medical record. One way in which this may be achieved is through recording this information on a Pharmaceutical Care Plan. Pharmaceutical Care is a model for pharmacy practice defined as 'the process through which a pharmacist co-operates with a patient and other professionals in designing, implementing and monitoring a therapeutic plan that will produce specific therapeutic outcomes'.²² A Pharmaceutical Care Plan is the component of the medical record through which this process may be documented.²³ An advantage of implementing a Pharmaceutical Care Plan would be to encourage direct contact between pharmacists and patients so as to provide more opportunities to add value to the medication review process through pharmacists' detailed knowledge of medicines and their capacity to identify potential medication-related problems.²⁴

The remaining six components of the ICD-10-AM Pharmacy Activity Codes are principally records of pharmacy workload that need not necessarily form part of the medical record. While the conduct of these activities may sometimes lead to relevant new information about the patient (such as an adverse drug reaction or a change to treatment resulting from therapeutic drug monitoring), such incidents would be separately recorded as an adverse drug reaction alert or a pharmacy intervention respectively. The conduct of the majority of these six activity codes represent checking and review activities that signify a clinical pharmacist's workload and do not need to be communicated to other health workers. An advantage of using the ICD-10-AM Pharmacy Activity Codes for this purpose is that they include definitions of what these activities represent, thereby encouraging standardisation. The lack of standardisation in clinical activity documentation has been noted as inhibiting practice-based research and identification of the pharmacist's contribution to patient care.^9,15,25

Records of pharmacy interventions should ideally be made in database format, as this aids classification and analysis. Many reports demonstrate the use of databases to record interventions, often in a bespoke form that impedes comparison between sites.^26,27 An alternative approach would be to utilize a database format that has common application in many practice settings, such as the Riskman database.²⁸ This would have the advantage of allowing the potential for comparing and benchmarking intervention rates and patterns between hospitals.

In recommending that pharmacists utilise a document such as a Pharmaceutical Care Plan to promote contact with the patient, to assist with medication reconciliation and to identify medication-related problems, we recognize that undertaking this activity for as many patients as possible represents a significant time commitment. Likewise, recording all interventions to treatment takes time. While we recognize that this represents a challenge to pharmacy management to meet this resource requirement, we believe that these activities provide the best means of identifying, resolving and documenting medication-related problems and the pharmacist's contribution to this task.

We acknowledged that using an observer may induce a 'Hawthorne effect', where subjects under observation tend to increase their work performance.^7,29 Countering this effect, however, was our use of an observer who was recognized as a peer by the pharmacists involved. Also, there was no performance 'rate' associated with the observations and the pharmacists were just advised that the observer was recording 'the activities that you perform'. In this study, we have utilized a single observer and did not conduct any inter- and intra-reliability testing.³⁰

During the periods of observation, the observer accompanied different pharmacists to different wards on different days, as our principal aim was to give the observer the opportunity to observe the greatest range of activities performed by clinical pharmacists as possible. The nature of the testing environment was thereby dynamic and the capacity to compare results obtained in a stable test setting was thereby limited. The ward environment (patient mix, the 'pace' of activity and other distractions) can vary from day to day, further limiting the utility of such exercises. We also recognize that observations that needed to be made by freehand note may have been under reported; our results may thereby have biased observations in favour of activities that could be readily selected from the data collection sheet.

In summary, we have identified the range of clinical activities performed by pharmacists in an Australian hospital inpatient setting. The records kept by pharmacists that could be used to provide a complete picture of clinical pharmacist activities should include a Pharmaceutical Care Plan, a database for intervention recording and a simple workload template. These findings have facilitated the commencement of an important study, viz: quantifying the relationship between the activities of clinical pharmacists and the patient's diagnosis related group (DRG). An improved understanding of this relationship is critical in supporting cost-effective delivery of pharmacy services and optimised patient care.

 

ACKNOWLEDGEMENTS

The authors wish to thank Ms Jayne Vella Johnson who carried out the role of observer in this study and who provided helpful advice in regard to the design of the data collection sheet. We also acknowledge the co-operation of the clinical pharmacists at The Northern Hospital and at Western Hospital Footscray during the observation phase of the study.

 

 

References

1. Cotter SM, Barber ND, McKee M. Survey of clinical pharmacy services in United Kingdom National Health Service hospitals. Am J Hosp Pharm 1994;51(21):2676-84.         [ Links ]

2. Reeder CE, Dickson M, Kozma CM, Santell JP. ASHP national survey of pharmacy practice in acute care settings--1996. Am J Health Syst Pharm 1997;54(6):653-69.         [ Links ]

3. Wilson SG, Tsui M, Tong N, Wilson DI, Chapman CB. Hospital pharmacy service provision in Australia--1998. Am J Health Syst Pharm 2000;57(7):677-80.         [ Links ]

4. Bright JM, Tenni PC. The Clinical Services Documentation (CSD) system for documenting clinical pharmacists' services. Aust J Hosp Pharm 2000;30(1):10-5.         [ Links ]

5. Jones AN, Benzie JL, Serjeant CS, Swan GT. Clinical pharmacy services: activity and cost analysis. Aust J Hosp Pharm 1984;14(2):50-3.         [ Links ]

6. Barsness FR, Trinca CE. Activity analysis and cost study of clinical pharmacists practicing in a university medical center: comparison with previously established criteria. Drug Intell Clin Pharm 1978;12(5):284-94.         [ Links ]

7. Jenkins D, Cairns C, Barber ND. How do ward pharmacists spend their time?: an activity sampling study. Int J Pharm Pract 1992;1:148-51.         [ Links ]

8. Larmour I, Allinson Y, Manser A, Spalding G, Brett K, Swan S. Clinical ward pharmacy survey - Victorian hospitals 1982. Aust J Hosp Pharm 1984;14(2):65-70.         [ Links ]

9. McLennan DN, Dooley MJ. National survey of clinical activity documentation practices. Aust J Hosp Pharm 2000;30(1):6-9.         [ Links ]

10. Australian Council for Safety and Quality in Health Care. Health reform - safety and quality action areas. Canberra: ACSQHC; 2004.         [ Links ]

11. Dooley MJ, McLennan DN, Galbraith KJ, Burgess NG. Multicentre pilot study of a standard approach to document clinical pharmacy activity. Aust J Hosp Pharm 2000;30(4):150-6.         [ Links ]

12. National Centre for Classification in Health. ICD-10-AM tabular list of procedures (MBS-extended). 1 July 1998 ed. Sydney: Faculty of Health Sciences, University of Sydney; 1998.         [ Links ]

13. Rascati KL, Kimberlin CL, McCormick WC. Work measurement in pharmacy research. Am J Hosp Pharm 1986;43(10):2445-52.         [ Links ]

14. Rutter PM, Brown D, Jones IF. Pharmacy research: the place of work measurement. Int J Pharm Pract 1998;6:46-58.         [ Links ]

15. Tenni PC, Hughes JD. National survey of clinical pharmacy services. Aust J Hosp Pharm 1996;26(4):416-27.         [ Links ]

16. Robinson NL, Stump LS. Benchmarking the allocation of pharmacists' time. Am J Health Syst Pharm 1999;56(6):516-8.         [ Links ]

17. Whitty JA, Green B, Cottrell WN. A study to determine the importance of ward pharmacists reviewing discharge prescriptions. Aust J Hosp Pharm 2001;31(4):300-2.         [ Links ]

18. Duguid M, Gibson M, O'Doherty R. Review of discharge prescriptions by pharmacists integral to continuity of care. -letter. Journal of Pharmacy Practice and Research 2002;32(2):94-5.         [ Links ]

19. Larmour I, Thomson WA, Tsui MK, Weeks GR. Introduction of Pharmaceutical Benefits Scheme reforms at three Victorian public health services. Journal of Pharmacy Practice and Research 2003;33(3):204-7.         [ Links ]

20. Gleason KM, Groszek JM, Sullivan C, Rooney D, Barnard C, Noskin GA. Reconciliation of discrepancies in medication histories and admission orders of newly hospitalized patients. Am J Health Syst Pharm 2004;61(16):1689-95.         [ Links ]

21. Cornish PL, Knowles SR, Marchesano R, Tam V, Shadowitz S, Juurlink DN, et al. Unintended medication discrepancies at the time of hospital admission. Arch Intern Med 2005;165(4):424-9.         [ Links ]

22. Hepler CD, Strand LM. Opportunities and responsibilities in pharmaceutical care. Am J Hosp Pharm 1990;47(3):533-43.         [ Links ]

23. Simioni D, Brien JE. Implementation of pharmaceutical care plans in a hospital ward. Aust J Hosp Pharm 1996;26(2):221-6.         [ Links ]

24. Wilcox C, Duguid MJ. The medication chart as an integral tool in the pharmaceutical care plan. Aust J Hosp Pharm 2001;31(4):268-74.         [ Links ]

25. McLennan DN, Dooley MJ, Brien JE. ICD-10-AM activity codes: an opportunity for standardisation. Aust J Hosp Pharm 1999;29(5):261-4.         [ Links ]

26. Lee AJ, Boro MS, Knapp KK, Meier JL, Korman NE. Clinical and economic outcomes of pharmacist recommendations in a Veterans Affairs medical center. Am J Health Syst Pharm 2002;59(21):2070-7.         [ Links ]

27. Johnson ST, Brown GC, Shea KM. Reengineering a pharmacist intervention program. Am J Health Syst Pharm 2002;59(10):916-7.         [ Links ]

28. Riskman (homepage on the internet) West Melbourne, Victoria. Software Design & Enhancement Pty Ltd est. 1987. (cited 2006 Dec 3) Riskman: incident notification (2 pages). Available from: http://www.riskman.net.au/framework_incident.html

29. Wickstrom G, Bendix T. The "Hawthorne effect"-what did the original Hawthorne studies actually show? Scand J Work Environ Health 2000;26(4):363-7.         [ Links ]

30. Wikipedia contributors St. Petersburg, Florida. Wikipedia, The Free Encyclopedia. updated Jan 4 2007 (cited 2007 Jan 13) Inter-rater reliability (1 page). Available from: http://en.wikipedia.org/wiki/Inter-rater_reliability