Introduction

Antimicrobial stewardships (AMS) programs are being progressively implemented in our hospitals. Their objectives are to improve clinical results, reduce those adverse effects associated with the use of antibiotics, including resistance, and guarantee a cost-effective therapy^1,2.

One of the main needs of AMS programs is to define indicators to measure the impact of their implementation. Among the indicators proposed to monitor the development of hospital AMS programs, there are structure indicators that describe the organization, composition, resources and tools necessary to carry out the necessary activities^{3 4 5-6}.

In order to meet the objectives of the AMS programs, indicators of clinical results have been developed, such as the reduction in the incidence of Clostridium difficile infection⁷, as well as for the mortality and hospital stay in selected infections⁸. Indicators for ecological results have also been proposed, shown as the reduction in the rates of bacterial resistance to antibiotics^8,9. These indicators are undoubtedly necessary, but difficult to evaluate, because their results are multifactorial variables without any direct or immediate link with an adequate use of antimicrobial agents.

On the other hand, process indicators have been defined, for example to measure the quality of use of antimicrobial agents in hospitals; their methodology is based on audits or prevalence surveys^4,10. However, their use is limited by the intensive labour, and the lack of homogeneity in the evaluation of prescription in critical aspects such as the evaluating staff, sample selection, parameters to be considered and their relative weight, and the degree of adaptation of each parameter^{11 12-13}. This leads to a high heterogeneity in the method and the results, and a significant burden of subjectivity and variability between observers in the final evaluation^12,13.

Monitoring the use of antimicrobial agents is one of the activities described in the AMS programs. Its role is to detect points susceptible to improvement actions and to evaluate the impact of interventions through the comparison between hospitals of the same level, and the analysis of time series in a single center, as well as to be used as support for the study of ecological impact^1,2.

Unlike the Primary Care setting, where the Spanish National Plan against Antibiotic Resistance (PRAN)¹⁴and the European Center for Disease Prevention and Control (ECDC)¹⁵have established indicators for the quality of use of antibiotics, these indicators have not been defined in the hospital setting.

The objective of this project was to identify some indicators of the use of antimicrobial agents in hospitals, through the comparative analysis of consumption data between hospitals of the same level, by the collective judgment of a group of experts.

Methods

A study was designed to identify indicators of the use of antimicrobial agents in hospitals, based on the comparative analysis of the consumption in hospitals of the same level, through the consensus of the group of experts following a modified Delphi methodology.

The development of the study is outlined inFigure 1. Initially, a coordinating committee, formed by the authors of this document, supported by the Group of Pharmacy Care for Infectious Diseases of the Spanish Society of Hospital Pharmacy (SEFH) and the Group for Healthcare- Related Infectious Diseases of the Spanish Society of Infectious Diseases and Clinical Microbiology, prepared a proposal of indicators based on their knowledge, experience, and literature review, considering criteria of efficacy, safety, ecological impact and cost (Table 1).

Figure 1. Stages and responsible for each phase of the study. 

Table 1. Indicators initially proposed by the Coordinating Committee 

AMS: antimicrobial stewardship; DDD/100OBD: number of defined daily dose per 100 occupied bed days; IV: intravenous; MRSA: methicillin-resistant Staphylococcus aureus; MSSA: methicillin-susceptible Staphylococcus aureus.

These indicators were defined using as source the data for hospital use of systemic antimicrobial agents, added by substance and administration route, according to the Anatomical Therapeutical Chemical Classification System and expressed in Defined Daily Dose (DDD) per each 100 occupied beds days (DDD/100OBD)¹⁶. The indicators have no established standard, but will acquire value through the comparison with hospitals of the same level.

Subsequently, the coordinating committee selected a panel of 21 Spanish experts with confirmed experience in the area of microbiology, infectious diseases, and antimicrobial therapy, formed by seven microbiologists, seven clinicians from infectious disease units, and seven hospital pharmacists who accepted to participate so that the proposed indicators would be submitted for their consideration. The board members are mentioned in the Acknowledgement section.

The RAND/UCLA method was used, which combines the Delphi and the Nominal Group techniques. This method consists in two evaluation rounds: the first one is conducted independently by each panel participant, and the second one in a face-to-face meeting¹⁷. In this study, the face-to-face meeting in the second round was replaced by a new individual evaluation.

The panel of experts was requested to qualify the relevance of each proposed indicator in four dimensions: Scientific evidence, efficacy and safety for patients, increase in microbial resistances, and cost for the health system.

The experts underwent two scoring rounds of a questionnaire sent by e-mail. There was a 1-to-9 scale, where 1 meant complete disagreement, 9 meant complete agreement, and 5 meant that it was uncertain.

In the first round, the panel of experts received the relevant bibliography and a questionnaire with the list of indicators, the definition of each, the formula for its calculation, and its justification for being proposed. The experts scored individually each one of the dimensions for the indicators. The experts were also encouraged to make comments about each indicator.

The coordinating committee evaluated the scores and comments by the panel, and prepared a second questionnaire, where the indicators were kept, modified or removed according to the first round scores, and issued an analysis of the results for each indicator in order to increase the elements of judgment for the experts.

In the second round, the modified questionnaire was sent, which the experts scored again, aware of the median and range of the first round scores, as well as the comments provided anonymously by the panel members and the analysis by the committee. The objective of this round was to give the experts the chance to review again their own evaluation, now with information about the evaluation by the other participants.

The scores given by the panel of experts to each one of the dimensions of each indicator were analyzed according to the procedure suggested in the RAND/UCLA method¹⁷.

Each dimension was classified as adequate, inadequate or uncertain, based on the median scores by the members of the group, and the degree of disagreement (Table 2).

Table 2. Definition of criteria for qualifying dimensions 

In order to define disagreement, the ratio between the Inter Percentile Range (IPR) and the Inter Percentile Range Adjusted per Symmetry (IPRAS) was used. An indicator was classified as with disagreement when the IPR was higher than the IPRAS. The IPR was calculated as the difference between the IPRAS = 2.35 + 1.5 · AI, where AI is the Asymmetry Index, which represents the distance between the IPR central point and the 5 value (central point in the scoring scale from 1 to 9)¹⁷.

The criterion to exclude an indicator to move on to the second round was that any indicator dimension was scored as inadequate.

The criterion to consider an indicator as relevant after the second round was that at least one dimension was scored as adequate, or at least 2 if one of them was cost, and that there were no dimensions scored as inadequate.

Results

The coordinating committee proposed a list of 14 potential indicators, based on the scientific evidence available, and on the experience and knowledge by the members of the committee (Table 1).

Table 1shows the description of each indicator, as well as the formula for its calculation, its direct or reverse relationship with good practice, and the justification for its proposal.

After the first round of scores, answered by the 21 participants of the panel of experts, the indicator for use of new bectalam agents was excluded, because it did not reach the sufficient criterion to move on to the second round. It obtained a median score of 8 in the cost dimension, 5 in resistance impact, 4 in scientific evidence and 3 in efficacy and safety. The majority of panel participants considered that the consumption of new bectalams could not be evaluated without considering the microbiological pattern of the hospital. Likewise, some members estimated that its low consumption could not be always considered good practice, and could even indicate excessive restriction causing clinical damage to patients, by not using the most adequate option. The rest of indicators obtained scores sufficient to move on to the second round.

The analysis by the committee of the comments and scores by the panel of experts in the first round led to the modification of two indicators for the second round: The macrolide consumption indicator was replaced by the ratio between the consumption of intravenous macrolides and intravenous respiratory fluoroquinolones, and the metronidazole consumption indicator was replaced by the ratio between the consumption of metronidazole and the sum of the consumption of carbapenems + piperacillin/tazobactam. The opinions of some board members, and the subsequent reflection by the committee, led to consider that the new definition of these two indicators showed their objectives in a better way.

After the second round, answered by all panel members, the indicators proposed reached the level of relevance according to the established criteria. Board members scored two indicators with the four dimensions as adequate, five indicators with three adequate dimensions, four indicators with two adequate dimensions, and two indicators with one dimension, different to cost, scored as adequate (Table 3).

Table 3. Results after the second round 

MRSA: methicillin-resistant Staphylococcus aureus; MSSA: methicillin-susceptible Staphylococcus aureus.

Discussion

We haven’t found any publication in literature establishing a set of indicators that allow to reach conclusions about the quality of the hospital use of antibiotics, exclusively based on consumption data.

All process indicators that evaluate the quality of use of antimicrobial agents in the hospital setting are based on specific prescription evaluations, which are cumbersome to conduct, with non-standard design and method, and with results depending on the criterion by the evaluator.

This study offers a new method to measure a series of good practices in the hospital use of antimicrobial agents based on a photograph of consumption, and by comparison between similar hospitals.

In the setting of primary care, and by using consensus methods, there has been a determination of indicators of the use of antibiotics based on consumption at national¹⁴and European¹⁵level. In both cases, there is no standard of re ference for the indicators established, but these acquire value by comparison with the data from other geographical areas.

In a similar manner to the initiatives mentioned in primary care, the indicators in our study draw upon the comparative analysis of the consumption data from hospitals of the same level. The more representative and reliable the consumption data are, and the higher the homogeneity by the portfolio of services and ecological pattern between hospitals compared, the more robust the results obtained.

The selection of indicators was based on prioritizing the consumption of the agents of choice in the main infections described in guidelines and protocols of reference, on reducing the use of antimicrobial agents in general, and of those with broad spectrum in particular, due to the impact on selection and spread of resistances, as well as on the application of measures for therapeutic adaptation and simplification, such as de -escalation strategy that replace empiric treatment by the narrowest possible spectrum agent, or the sequential therapy with the switch from intravenous to oral administration.

There is enough evidence proving that the reduction in the use of antimicrobial agents, particularly those with broader spectrum, will offer benefits for the health system, such as savings in direct costs and the potential reduction in microbial resistances, without affecting the clinical course of patients^8,9,18,19. The benefit in the application of de-escalation practices^20,21and sequential therapy^22,23has also been documented.

Our study has various limitations. The indicators selected were developed by consensus from a multidisciplinary panel of professionals, clinicians who are expert in infectious diseases, microbiologists and hospital pharmacists. Even though this structure is considered optimal for the development of antibiotic policies in hospital care, it only shows the subjective opinion and knowledge by a group of experts, without objective validation or confirmed scientific evidence. The use of multicriteria decision analysis for scoring the quality indicators can also be questionable, but this strategy is necessary in order to address the different dimensions in the evaluation of antimicrobial use.

However, the greatest limitation concerns the definition itself of the indicators, and the fact that these depend exclusively on consumption data. Numerous omments by panel members about various indicators referred to the need of complementing the antimicrobial consumption data with the local pattern of microbial resistances, the portfolio of services by each hospital, and the clinical results obtained, regardless of conducting the comparison between hospitals with the same level of complexity.

Even though these indicators have no absolute reliability in order to measure the quality of use of antimicrobial agents, and require individual interpretations of the results in each case, according to the particular characteristics of each hospital, these indicators are necessary, because it is required to measure the use of antimicrobial agents, they are feasible because they are easy to implement, and objective because they are not dependent on the subjective evaluation of an observer.

The indicators defined will be incorporated to the platform in the SEFH website, for entering antimicrobial consumption data based on pharmacy unit dispensing. The centers which enter their consumption data voluntarily will have access to the information about the relative value of the indicators. The analysis of the results of the indicators during the next years will be used for their validation and to confirm their utility.

The information provided by applying the indicators can be very useful in order to allow hospital AMS teams to detect any potential problems with the use of antimicrobial agents, and to help guide their efforts to implement actions of improvement, as well as to value the impact of the measures conducted.