SciELO - Scientific Electronic Library Online

vol.41 número1Coste-eficacia del pemetrexed en combinación con cisplatino como tratamiento de primera línea en pacientes con cáncer de pulmón avanzado no microcítico ni escamoso en EspañaSeguimiento farmacoterapéutico en pacientes en tratamiento con agentes biológicos por enfermedades sistémicas inflamatorias crónicas: acuerdo entre farmacéuticos hospitalarios para la recopilación estandarizada de un conjunto mínimo de datos índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados




Links relacionados

  • En proceso de indezaciónCitado por Google
  • No hay articulos similaresSimilares en SciELO
  • En proceso de indezaciónSimilares en Google


Farmacia Hospitalaria

versión On-line ISSN 2171-8695versión impresa ISSN 1130-6343

Farm Hosp. vol.41 no.1 Toledo ene./feb. 2017 


Reduction in alert fatigue in an assisted electronic prescribing system, through the Lean Six Sigma methodology

M.ª Jesús Cuéllar Monreal1  , Jorge Reig Aguado1  , Isabel Font Noguera1  , José Luis Poveda Andrés1 

1 Pharmacy Department. Hospital Universitari i Politècnic La Fe, Valencia. Spain



To reduce the alert fatigue in our Assisted Electronic Prescribing System (AEPS), through the Lean Six Sigma (LSS) methodology.


An observational (transversal) and retrospective study, in a general hospital with 850 beds and AEPS. The LSS me-thodology was followed in order to evaluate the alert fatigue situation in the AEPS system, to implement improvements, and to assess outcomes. The alerts generated during two trimesters studied (before and after the intervention) were analyzed.

In order to measure the qualitative indicators, the most frequent alert types were analyzed, as well as the molecules res-ponsible for over 50% of each type of alert. The action by the prescriber was analyzed in a sample of 496 prescriptions that generated such alerts. For each type of alert and molecule, the-re was a prioritization of the improvements to be implemented according to the alert generated and its quality. A second sur-vey evaluated the pharmacist action for the alerts most highly valued by physicians.


The problem, the objective, the work team and the project schedule were defined. A survey was designed in order to understand the opinion of the client about the alert system in the program. Based on the surveys collected (n = 136), the critical characteristics and the quanti/qualitative indicators were defined.

Sixty (60) fields in the alert system were modified, correspon-ding to 32 molecules, and this led to a 28% reduction in the total number of alerts. Regarding quality indicators, false positive results were reduced by 25% (p < 0.05), 100% of those alerts ignored with justification were sustained, and there were no significant differences in user adherence to the system. The project improvements and outcomes were reviewed by the work team.


LSS methodology has demonstrated being a valid tool for the quantitative and qualitative improvement of the alert system in an Assisted Electronic Prescription Program, thus reducing alert fatigue.

KEYWORDS: Electronic prescription; Clinical decision support systems; Medication alert system; Alert fatigue; Lean Six Sigma; Quality improvement

Contribution to scientific literature

The objective of this original article was to reduce fatigue alert in our Assisted Electronic Prescription System, through the optimization in the number and quality of alerts, and the standardization of the pharmacist action when faced with the different types of alerts.

The fatigue alert we have detected in our daily practice is something frequent in Assisted Electronic Prescription (AEP) alert systems. It has been confirmed that many deficiencies are still identified, in spite of the wide experience and the literature available about the implementation of these systems. Moreover, there is a limited number of studies about their efficacy in terms of the relevance and quality of the information offered, as well as about the response of users to this information, and all this can compromise the efficacy and safety of the system.

We believe that the reason for both the originality and the relevance of these outcomes is the methodology followed, Lean Six Sigma, which presents the characteristic of being supported by a work team, that is to say, it requires consensus by users; and most of all, improvements are introduced according to the “Client Voice” or user opinion. Regarding this, we consider that there was a enough level of participation (approximately 10% of hospital physicians).

This evaluation of the alert system by a high number of users, together with the analysis of the quality of the most frequent types of alert and the improvements implemented, will gradually generate valuable information about the real utility of alert systems in AEP programs.


Assisted Electronic Prescription Systems (AEPS) represent, in the setting of safety improvement for patients, an important support tool for making clinical decisions at the time of prescription1.

These systems have had a significant development in recent years; therefore, besides a wide range of clinical information, the majority of them include a potent system of safety alerts. However, there are few studies on their efficacy, and their clinical benefits for patients have not always been demonstrated2,3,4,5.

It has been described that the alerts generated by an AEP system are ignored in up to 49-96% of cases3. This shows a problem in user adherence to recommendations; one of the main causes for this is the high number of alerts generated (between 34 and 90% of prescriptions); moreover, these are not always relevant, specific, and easy to manage. All this will lead to the development of “fatigue”6.

Peterson and Bates7 describe “alert fatigue” as “the state of mind resulting from receiving too many alerts, which are time-consuming and require too much mental energy, and that can be the cause for some clinically relevant alerts to go unnoticed among other non-relevant alerts”. That is to say, there is a desensitization of the user, who ignores the alerts that are not important as well as those that are important, thus compromising the objective of these alerts, which is patient safety.

Our AEP system features various information and safety levels. Firstly, it provides basic information that will make prescription easier. Secondly, it conducts a basic check of the prescription, according to a knowledge base of reference (defined regimens, maximum dosing, maximum number of treatment days, interactions, contraindications and duplications). And finally, it includes an advance check that integrates relevant patient data, such as history of allergies and certain clinical conditions, even decision algorithms, such as the renal impairment module for dose adjustment in those medications that require it, and dosing based on body weight and/or age for paediatric patients. All this will generate alerts that appear interactively and/or as a report at the end of the prescription (this option can be set up by each user based on their preferences and experience). But the prescription is never blocked, and there is no obligation to justify the decision to ignore the alert; and therefore, there is no feedback that would be required for updating the system1.

Despite this, and the fact that there has been a selection of clinically significant alerts, and these have been customized according to the clinical situation of the patient and adverse event severity, the presence of alert fatigue has been detected in our daily practice. Therefore, it is necessary to improve the efficiency of the alert system, reducing the workload but without any negative impact on patient safety.

These improvements require consensus by users and clinical committee; they must be introduced cautiously and their results must be assessed. Therefore, this project has been developed following the Lean Six Sigma (LSS) methodology; its use is not widespread in the healthcare setting, but it has already provided some successful experiences8. Indeed, this is a scientific methodology for continuous quality improvement, supported by a work team, and understanding client needs. The key idea is to apply Lean tools to eliminate any unnecessary stages without any value (waste), and reduce the cycle times (efficiency), as well as the Six Sigma Techniques in order to reduce variability and deficiencies (effectiveness and safety)9.

The objective of the present article is to optimize the number and quality of alerts in our AEP program, to standardize the action by pharmacists when faced with different types of alerts, and to increase user satisfaction.


Ethical aspects

The project was authorized by the Head of Department (“Champion”, according to the LSS methodology). Electronic records did not include any identification data for patients or healthcare professionals.


An observational, transversal and retrospective study. Improvements were implemented and outcomes were assessed by deploying the stages of the DMAIC (Define, Measure, Analyze, Improve, Control)9 cycle of the LSS methodology.


A general hospital of reference with 1,000 beds, 850 of them with AEP, with over 1,000 licences for prescribing physicians, and 1,450 pharmaceutical products available in the system for prescription.

Population and sample

The study included the alerts generated during two trimesters (before and after the intervention), obtained from the AEP system. For the qualitative analysis of the project, 496 prescriptions in total were reviewed.

Schedule of the DMAIC cycle

The LSS project was conducted between May, 2014 and May, 2015. It was structured on the DMAIC cycle stages.

In the Define Stage (May-June, 2014), the problem, objective, work team and project schedule were defined. The Work Team was formed by five users of the AEP system (4 pharmacists and 1 physician), an external observer physician, a pharmacist for methodological support, and the pharmacist managing the AEP system database, as group liaison.

According to the LSS methodology, the first step is learning the “Client Voice” about the alert system in the program. For this aim, a survey was designed with six questions about the quantity and quality of alerts, as well as their attitude towards them; this survey was distributed to all physician and pharmacist users (specialists and residents). The survey was completely anonymous, although it included data about the clinical department and the professional category.

Based on the analysis of Client Voice, the “critical characteristics” of the project were determined (LSS methodology), and those quantitative and qualitative indicators to be measured were defined (Table 1).

Table 1 Critical characteristics derived from Client Voice and quantitative and qualitative indicators 

In the “Measure, Analyze” Stages (July-September, 2014), those alerts generated during the first trimester of 2014 were quantified, as well as the number of hospitalized patients and the number of lines of prescription modified during this period.

In order to measure the qualitative indicators, the most frequent types of alert were analyzed, as well as the molecules responsible for over 50% of each type of alert. A sample of the prescriptions generating these alerts was analyzed, from the perspective of alert relevance (whether adequate or not), and of the clinical action taken for these alerts (acceptance or not) (Figure 1).

Figure 1 Alert Classification and Definition of Quality Indicators 

The pharmacist action for the alerts most valued by physicians was assessed through a second survey.

In the “Improve” Stage (October-December 2014), it was considered to eliminate and modify those alerts that were no valuable, according to the predefined criteria for a good alert system3,6,10. For this aim, the improvements to be implemented were prioritized, and the work team reached a consensus. The initial objective established was the reduction by half of the alert load.

With the aim to reduce the variability in pharmacist action for different types of alert, a series of recommendations were made to the Pharmacy Team.

In the “Control” Stage (January-May, 2015), the indicators were reassessed, based on data extracted from the AEP system for the first trimester in 2015, once the improvements had been implemented.

There was an estimation of the benefit in the reduction of the number of alerts, based on the assumption that the time required to read an alert is 1.7 seconds.

Statistical Analysis

The two-proportion z-test.was used for the statistical analysis of the outcomes. The predetermined level of significance was 5%. The analysis was conducted with the statistical analysis G-Stat 2.0 program.


Survey (Client Voice)

In total, 136 surveys were collected from 24 clinical departments, both medical and surgical. A 69% of the participants were specialists, and 31% were residents. The results of the survey are shown in Table 2.

Table 2 Results of the survey (Client Voice) on the alert system (n = 136) 

Ẋ = arithmetic mean standard deviation

These results were translated into the critical characteristics of the project, and an indicator was defined for each one, with its criterion and measurement (Table 1).

Initial situation

During the first trimester in 2014, there were 47,994 different alerts; their rate of distribution by type of alert is shown in figure 2.

Figure 2 Percentage distribution of the type of alerts at the initial situation 

The results of the quantitative indicators appear in Table 3.

Table 3 Quantitative Indicators 

For estimating the quality indicators, there was a selection of 71 out of 971 molecules (7.3%), responsible for over half of the alerts generated for each type of alert (58%). After the assessment of each type of alert, these could be classified into two groups:

Alerts caused by software limitations: therapeutic duplication, dispensing conditions, and allergies.

Alerts that represented a problem in the quality of the alert or the action by the user, such as those for renal impairment, geriatrics, maximum doses, or interactions. For their analysis, 269 prescriptions were reviewed, and the predefined quality indicators were estimated (Table 4).

Table 4 Quality Indicators 

*Values weighted per sample size (n)

Table 5 shows the results of the survey on pharmacist action for those alerts valued as the most relevant by physicians; this survey was distributed among 19 Pharmacists (5 residents, 14 specialists).

Table 5 Survey on Pharmacist action for the alerts most valued by physicians (n = 19) 

Improvement Actions

A potential improvement was suggested for each type of alert and molecule, according to the alert generated and its quality. The improvements to be implemented were prioritized according to their feasibility and efficiency, there was a risk evaluation, and consensus was reached in the work team about the measures to be taken. Sixty (60) fields were modified in the alert systems, corresponding to 32 molecules (Table 6).

Table 6 Improvement actions for the alert system 

In order to reduce the variability of Pharmacist action when faced with an alert, the work team agreed by consensus on a series of recommendations, which were distributed among Pharmacists (Table 6).

Final situation

After implementing the improvement actions, there was a reevaluation of the quantity and quality indicators related to the three months after said implementation. The results of this second evaluation, in comparison with the initial values, are shown in Tables 3 and 4. An important reduction in quantity indicators can be observed, although it is inferior to the 50% initially predefined.

Regarding quality indicators, false positive results were reduced by 25% (difference in proportions = 0.25, CI 95% = 0.163-0.337, p < 0.05), and the number of inadequate alerts was reduced; because none of the alerts left (due to limitations in the program) were accepted by the user, the 100% of alerts ignored with justification was sustained, and there were no significant differences either in terms of user adherence to the system.

Economic impact

The estimated benefit consists mainly in saving time that can be used for other tasks. The 28% reduction (13,313) in the number of alerts generated allows to estimate that 4.3 hours per month are saved in prescription and validation time. If we also consider that each alert is visualized for a mean of 6 times throughout patient’s treatment follow-up, and that each time it is read again, this saving could reach 25.8 hours per month.

End of the project

The improvements and project results were reviewed and discussed by the work team, and then communicated to healthcare professionals through the hospital intranet by computer graphics (Figure 3).

Figure 3 Computer graphics published in the hospital intranet to communicate the project outcomes 


The ideal AEP system would provide information to the user that is perceived as very useful, and achieve the adequate balance between the number of alters and the guarantee of patient safety. Our alert system already features the essential alert categories described in literature, and those characteristics defining a good alert system, such as specificity, sensitivity, clear and justified information, stating severity and the action required, and no interruption of the work flow (only the most important alerts appear in an interactive format).11 However, there is room for improvement. And this article is an approach to real practice, based on user opinion about the alert system, in terms of quantity and quality, in order to suggest measures to reduce alert fatigue.

According to the survey, the majority of prescribers (93%) claim that they read the alerts, but only 30% declare that they always follow their recommendations, which shows system alert fatigue. Prescribing physicians become sceptic regarding the validity of the system, trusting their own knowledge or assumptions, and/or underestimating the potential adverse effects of medications10. In fact, very few users (6%) acknowledge that they have detected some negative consequence by not following the recommendations. This coincides with the results of previous studies, where adverse effects have been detected only for 2.3 to 6% of those alerts ignored3.

Regarding the total number of alerts, even though it was not considered excessive by users, it has been reduced by 28%. Therefore, from visualizing an alert for each 3.6 lines modified, there is now an alert per each 5.1 lines modified. And all this without any negative impact on patient safety, because there is still a 100% proportion of alerts ignored with justification, and only those alerts ignored without justification represent a threat from the point of view of safety.

The most valued alert is that regarding drug allergies; and practically all participants (89%) answers that they would like the line of treatment to be blocked when this type of alert appears. In our case, even though there is high sensitivity, it only generates 2% of the total number of alerts, but still presents false positive results, mainly due to an incorrect record of the history of patient allergies (real allergy vs. lack of tolerance) and/or when the alert is based upon potential cross-sensitivities12,13. The improvement consisted in standardizing pharmacist action at the level of allergy coding and intervention to be conducted.

The renal impairment alert is the third most valued alert, and responsible for 25% of those alerts recorded. The information available has been reviewed and updated, in order to improve its perceived utility.

Four molecules were practically responsible for geriatric alerts, mainly regarding a gradual increase in dose and monitoring in geriatric patients when, in the majority of cases, these medications were already part of their chronic treatment; therefore, it was decided to eliminate these recommendations.

Regarding maximum doses, low adherence has been detected, which can only be corrected through pharmacist intervention.

In terms of interactions, it is known that the sensitivity and specificity of interaction alerts have a high impact on the quality and number of alerts presented. It is recommended to include only those which are clinically relevant and require some action14. The module of interactions has been reviewed and updated, and pharmacist action has been improved. Algorithms must still be introduced that combine the prescription of one or more drugs with lab test data and their adequate range; this would increase the specificity of these alerts, because one of the most frequent reasons to ignore them is the intent to monitor the patient13,15,16. The incorporation of these highly specific algorithms would also allow to alert about other contraindications in general, which is information highly valued by physicians, according to this survey.

All this has achieved a 25% reduction in false positive results; that is to say, there has been an increase in the utility of the alert systems (PPV). But alert fatigue is a complex situation, associated with a human component which determines that the user will read the alert and value its level of utility at the time of making a clinical decision. Therefore, the real impact of reducing the volume of alerts is still to be determined16,17. And even though the number of alerts has been reduced, no improvement has been achieved regarding the level of adherence by the user, who continues eliminating them instinctively. In order to improve this adherence, it will probably be necessary to inform users about these improvements; and this has been done by publishing computer graphics in the hospital intranet, and allowing for a period of adaptation.

Regarding the personal alert setup (question F), very few medical users remember that they can change their profile (after their initial training). Even though this is one of the factors most frequently mentioned in literature as an example of how to optimize the alert systems, in the study by Jung et al. it appears as the aspect least valued by physicians18. And according to the opinion of users in our centre, this is not an option that they consider essential for the improvement of the system.

One step further in order to improve process efficiency has consisted in reviewing the variability in pharmacist action when faced with the alerts. Through a training session we have standardized our action, defining how, when, who, and in which circumstances there must be an intervention.

An important aspect in the future will be to establish the relationship between making clinical decisions by alert and the quality of patient care or health outcomes. In fact, an evolution in AEP system information towards health outcomes is already being demanded14,17.

As a conclusion, there has been a quantitative and qualitative improvement in the alert system, following the LSS methodology. However, continuous maintenance will be required, with iterative refining and function monitoring, in order to ensure an optimal function of these tools for supporting clinical decision, as well as trust by the user and their ability to react when faced with the information offered,

The authors declare that there is no conflict of interests, and that there has been no funding whatsoever for this project.


To Asunción Albert, Manuela Martínez, Emilio Monte, Amparo Peris and Carmen Planells for being part of the work team for the project, and their commitment and advice.


Cuéllar MJ, Poveda JL. Soporte a la decisión clínica en prescripción electrónica. Ayudas para una mejor prescripción. El Farmacéutico de Hospitales 2004;159: 19-28. [ Links ]

McCoy AB, Waitman LR, Lewis JB, Wright JA, Choma DP, Miller RA, Peterson JF. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc. 2012;19:346-52. [ Links ]

Van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13 (2):138-47. [ Links ]

Ojeleye O, Avery A, Gupta V, Boyd M. The evidence for the effectiveness of safety alerts in electronic patient medication record systems at the point of pharmacy order entry: a systematic review. BMC Med Inform Decis Mar. 2013; 13: 69. [ Links ]

Wolfstadt JI, Gurwitz JH, Field TS, Lee M, Kalkar S, Wu W, Rochon PA. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008; 23 (4): 451-8. [ Links ]

Van der Sijs H. Drug safety alerting in computerized physician order entry. Unraveling and counteracting alert fatigue. Rotterdam: Erasmus University; 2009. [ Links ]

Peterson JF, Bates DW. Preventable medication errors: identifying and eliminating serious drug interactions. J Am Pharm Assoc (Wash). 2001; 41:159-60. [ Links ]

Font I, Fernández MJ, Ferrer AJ, Balasch S, Edo MD, Poveda JL. Mejora del proceso farmacoterapéutico del paciente hospitalizado mediante la metodología Lean Seis Sigma. Rev Calid Asist. 2013; 28 (6):370-80. [ Links ]

Gupta P. The Six Sigma performance handbook. A statistical guide to optimizing results. 1ª ed.New York, Ed.Mcgraw-Hill; 2005. [ Links ]

Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA Annu Symp Proc. 2009; 2009: 548-52. [ Links ]

Phansalkar S, Edworthy J, Hellier E, Seger DL, Schedlbauer A, Avery AJ, Bates DW. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc. 2010; 17: 493-501. [ Links ]

Carspecken CW, Sharek PJ, Longhurst C, Pageler NM. A clinical case of electronic health record drug alert fatigue: consequences for patient outcome. Pediatrics 2013; 131: e1970-3. [ Links ]

Nanji KC, Slight SP, Seger DL, Cho I, Fiskio JM, Redden LM, Volk LA, Bates DW. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014; 21 (3): 487-91. [ Links ]

Clinical Decision Support workshop meeting summary. Office of the National Coordinator for Health Information Technology. Department of Health & Human Services. August 25-26, 2009. [ Links ]

Eppenga WL, Derijks HJ, Conemans JM, Hermens WA, Wensing M, De Smet PA. Comparison of a basic and an advanced pharmacotherapy-related clinical decision support system in a hospital care setting in the Netherlands. J Am Med Inform Assoc. 2012;19:66-71. [ Links ]

Phansalkar S, Van der Sijs H, Tucker AD, Desai AA, Bell DS, Teich JM, Middleton B, Bates DW. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc. 2013; 20: 489-93. [ Links ]

Coleman JJ, Van der Sijs H, Haefeli WE, Slight SP, McDowell SE, HM Seidling, Eiermann B, Aarts J, Ammenwerth E, Ferner RE, Slee A. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Medical Informatics and Decision Making 2013, 13:111. [ Links ]

Jung M, Riedmann D, Hackl W, Hoerbst A, Jaspers MW, Ferret L, Lawton K, Ammenwerth E. Physicians’ perceptions on the usefulness of contextual information for prioritizing and presenting alerts in computerized physician order entry systems. BMC Medical Informatics and Decision Making 2012; 12: 111. [ Links ]

Aronoff GA, Bennett WM, Bern JS. Drug Prescribing in Renal Failure: Dosing Guidelines for Adults and Children, 5th ed. Philadelphia, PA: American College of Physicians; 2007. [ Links ]

Mensa J, Gatell JM, García-Sánchez JE, Letang E, López-Suñé E, Marco F. Guía de Terapéutica Antimicrobiana. 14 ed. Barcelona, Ed. Antares; 2014. [ Links ]

AGS updated Beers Criteria for potentially inappropriate medication use in older adults. New York. American geriatrics society, 2012. Actualización 2014. [ Links ]

Drug Interactions. Micromedex. Consulta en: Mayo 2015. [ Links ]

Received: December 16, 2015; Accepted: September 29, 2016

* Autor para correspondencia. Correo electrónico: (M.ª Jesús Cuéllar Monreal).

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License