REVISIÓN

 

Measuring the performance of health care services: a review of international experiences and their application to urban contexts

Medir el desempeño de los servicios de salud: revisión de las experiencias internacionales y su aplicación a contextos urbanos

 

 

Anna García-Altés ^a Lauriane Zonco ^b Carme Borrell ^c Antoni Plasència^c,  for the Barcelona group on the performance of health care services

^a Agència de Salut Pública de Barcelona, Barcelona, Spain
^b Department of Health and Medical Services, Dubai, United Arab Emirates
^c Direcció General de Salut Pública, Departament de Salut, Barcelona, Spain.

Dirección para correspondencia

 

 

---

ABSTRACT

Background: The objective of performance assessment is to provide governments and populations with appropriate information about the state of their health care system. The objective of this paper is to present the most recent developments in performance assessment and their application in urban contexts.
Methods: Literature review in PubMed (1970-2004). We identified additional papers and grey literature from retrieved references.
Results: Performance assessment initiatives were identified in Australia, Canada, the United Kingdom, and New Zealand. The World Health Report 2000 is one of the best known examples of a transnational approach to performance assessment.
Conclusion: The best developed initiatives to date are those that define precise categories, criteria and indicators with which to analyse and assess health care systems, based on a solid conceptual framework. Performance assessment fits perfectly in urban contexts, as it is a useful tool for designing and monitoring policies, assessing the quality of the services provided, and measuring the health status of city dwellers. Barcelona and Montreal are currently collaborating together on a project to assess the performance assessment of their respective health care services.

Keywords: Outcome and process assessment. Health care systems. Health status.

---

RESUMEN

Objetivos: El objetivo de la evaluación del desempeño es dar información apropiada a los gobiernos y las poblaciones sobre el funcionamiento de su sistema sanitario. El objetivo de este artículo es presentar los desarrollos más recientes en la medida del desempeño y su aplicación en contextos urbanos.
Métodos: Revisión de la bibliografía en PubMed (1970-2004). Se identificaron artículos adicionales y bibliografía gris y de las referencias de los artículos seleccionados.
Resultados: Se identificaron iniciativas de medida del desempeño en Australia, Canadá, Reino Unido y Nueva Zelanda. La iniciativa transnacional del Informe sobre la salud en el mundo 2000 de la Organización Mundial de la Salud (OMS) es una de las más conocidas.
Conclusiones: Las iniciativas más desarrolladas en la actualidad son las que definen categorías, criterios e indicadores para analizar y evaluar los sistemas sanitarios, basándose en un marco conceptual sólido. La medida del desempeño encaja perfectamente en los contextos urbanos, ya que es una herramienta útil para diseñar y hacer un seguimiento del nivel de consecución de las políticas en marcha, medir la calidad de los servicios disponibles y el estado de salud de los habitantes de una población. Barcelona y Montreal están trabajando en un proyecto de colaboración para la evaluación del desempeño de sus servicios sanitarios.

Palabras clave: Evaluación de resultados y de proceso. Sistemas sanitarios. Estado de salud.

---

 

 

Introduction

Performance can be defined as the set of activities and programs that are carried out in order to achieve a series of previously established objectives and goals¹. In other words, in a health care system, performance implies a strategic orientation, a specific type of organization, and a health system structure that facilitates the fulfilment of its mission of attaining the goals established by the government and society as a whole¹.

In recent years, an increasing amount of work and attention have been devoted to measuring performance within health systems with the ultimate objective of improving them. The driving forces behind this trend are rooted in the current characteristics of the environment in which health care takes place and the concerns of governments and service users². Constraints on resources and particularly spending, growing public expectations and concerns about safety, quality and equity all increase this pressure and demands for accountable health care systems.

The objective of performance assessment is to provide governments and users with appropriate information about the state of their health care system³. In an ideal world, this information would in turn be used to re-design these health care systems in the most effective, efficient, and equitable manner and to the satisfaction of all of those involved in this system (consumers, professionals, providers, managers, and governments).

The Pan American Health Organization has identified some of the duties of any relevant performance assessment¹:

- To monitor the main factors that detemine of health trends.
- To compile an evidence-base for the relationship between the design of a particular health system and its performance and thus provide policy makers with the tools required to develop more effective systems.
- To help determine priorities for health care interventions and to contribute to their design, management, and follow-up.
- To provide analysis of and share information about different experiences in order to make better and more effective evaluations.
- To assess progress in relation to key health system goals.

Urban contexts have all the elements required to undertake such an initiative. Big cities have a series of social problems that are largely a consequence of their development and growth: low income families, unemployment, immigration, inequalities, small social networks, etc. Existing health problems tend to be a consequence of these social conditions⁴. In many cities, the available public health and health care services try to address these issues with specific policies and programs. Urban contexts could be seen as micro-environments in which to try out such initiatives. They are much more focused than the broader system and allow us to measure the achievement of specific health goals and the effectiveness of programs in situ.

The objective of this paper is to provide examples of the practical application of performance assessment. We have sought to do this by presenting the most recent developments in this field, citing its use in a wide variety of national and international organizations and institutions, identifying the strengths and weaknesses of different applications, and discussing their applicability to urban contexts.

 

Methods

We searched references published in PubMed between 1970 and 2004 using performance, measurement, health, system, and policy as key words. From the literature retrieved, we selected papers dealing with whole system performance assessment initiatives that had actually been carried out in different countries, but excluded quality registers and accreditation requirements. We also identified a number of additional papers and some grey literature from the references.

Performance assessment initiatives were identified in Australia, Canada, the United Kingdom, and New Zealand and we also found references to the trans-national performance assessment initiative of the World Health Report. We described these performance assessment initiatives starting at the international scale and then listing country-specific experiences in alphabetical order. We assessed the applicability of the different initiatives reviewed to the urban context, and evaluated their ability to reflect the nuances of their particular settings, and to assess the health status of the inhabitants of big cities and the public health and health care services available to them.

 

Results

The World Health Report 2000

The objective of The World Health Report 2000 was to determine whether a particular health system was performing as well as it could. This was assessed in terms of its ability to attain the three intrinsic goals of any health system. According to the World Health Organization (WHO) these goals involve: offering improved health for everyone, the system being responsive to the expectations of its users, and it being fair in terms of the financial contributions involved⁵.

Five indicators were used to measure this performance. The first was the level of health, measured by disability-adjusted life expectancy (DALE). Briefly summarized, DALE is a measure of life expectancy adjusted to take into account the time lived with a disability. DALE is estimated from the percentage of the population surviving to each age, calculated from birth and death rates; the prevalence of each type of disability at each age; and the weight assigned to each type of disability, which may or may not vary with age. DALE is therefore the number of years of good health that an average baby born in a given country at a certain point in time could reasonably expect to enjoy.

The second indicator focused to how health is shared throughout the population: it relates to inequality in health and health care and was measured by an index based on child mortality, which can be applied for all countries with complete birth history data. Income inequality, education inequality and access to the health system were also analysed to determine their contribution to child survival inequality. Results showed that the main factor that explained health inequality was access to the health system.

The overall responsiveness of the health system and its accessibility to the population constituted our third indicator of performance. These parameters reflect respect for people's rights and the extent to which the service provided is user oriented. Respect for people includes respect for the dignity of the individual, confidentiality, and autonomy; and the right of the individual to participate in choices relating to his/her own health.

The fourth indicator, client orientation, was assessed in terms of prompt attention, the quality of basic amenities, access to social support networks, and being able to choose the service provider. Measurement was based on a survey involving 1,006 respondents from 125 countries, half of whom were WHO staff. Finally, the fifth indicator was related to the fairness of financial contributions, which was measured by an index based on the proportion of expenditure spent on health care.

Weights were then assigned to the indicators, which were again based on a survey to key respondents. A weighting of 50% was assigned to overall health outcome, with 25% being assigned to level of health and 25% to health distribution. Half of the weighting assigned to health distribution, 12.5% of the total, was attributed to the level of responsiveness, while the other half was assigned to its distribution. Finally, the remaining 25% was attributed to fair financing. These weighted scores were then used to create a single composite index.

Australia

In Australia, extensive benchmarking has been carried out at all levels -local, regional, state, and national- and in both the private and public sectors. This has accompanied an increasing appreciation of the importance of performance measurement and the value of benchmarking within the health sector. The Australian Health Ministry set up the National Health Performance Committee (NHPC) in August 1999 in order to continue the work of the National Health Ministers' Benchmarking Working Group⁶. This Committee is responsible for developing and maintaining a national performance measurement framework for the health system, promoting benchmarking to help improve the health system, and providing information on national health system performance. This information will have a direct impact on the development of future health policy and funding, and will guide decisions concerning where to make investments in order to best improve the long term health and well being of the Australian population.

Government involvement in the health system aims to efficiently and effectively protect and restore the health of the community by:

- Promoting health and health care.
- Determining causes of ill health and reducing community exposure to negative lifestyles and environmental risks.
- Preventing and detecting illness through the provision of services that can foster improved health outcomes at relatively low cost.
- Caring for the sick through the use of appropriate intervention services.
- Providing appropriate health care services which recognize and respect cultural differences.
- Providing equitable access to these services.

The NHPC has established strategic plans for performance indicator development and is supported by working groups that deal with specific issues and areas and also provide technical advice and assistance. The Committee has set itself 3 key goals:

- To extend the national performance indicator framework to cover more than acute inpatient services. This involves developing indicators for the overall performance of the health system and also for other services such as community health, general practice and public health.
- To establish good links with other professionals involved in the vast range of work being undertaken in the field of performance indicator development across the nation and to share in and take advantage of their knowledge.
- To improve the timeliness for reporting information on performance.

The NHPC is currently developing a national health performance framework. This will facilitate performance reporting at the national level and also at the state and local levels as well as for specific target populations or populations judged to be at risk. The future work of the NHPC will include compiling a database of health performance indicators relevant to: dimensions of performance, health priority areas and interventions, health outcomes, health determinants and health system infrastructure.

Canada

In 1998, over 500 people -health administrators, researchers, caregivers, government officials, health advocacy groups, and consumers- were brought together to identify health information needs. One of their priorities was to collect comparable quality data on key health indicators for health and health services. In response, the Canadian Institute for Health Information (CIHI) and Statistics Canada launched a collaborative process to identify what measures should be used to report on health and the health system, and how to share this information with the Canadian population⁷.

The main goal of the Health Indicators project is to help health regions to monitor progress, to maintain and, if possible, improve the health of the population and the health system. It is aimed to do this through the provision of quality, comparative information on:

- The overall health of the population and how this compares from region to region, within specific provinces and the country as a whole, and how this has changed over time.
- The major non-medical determinants of health in each region.
- The health services received by the residents of each region.
- Characteristics of each community and health system.

The health indicators selected had to be relevant to established health goals, based on standard definitions and methods, and broadly available, as they had to be applicable throughout Canada at the regional, provincial and national levels. Consultations continued with provincial and regional health authorities to refine and expand the initial list of indicators, which were confirmed at a consensus conference. As data are compiled, the provincial and regional authorities are also involved in the verification process.

The United Kingdom

The National Health Service (NHS) has behind it a long history of performance measurement and this has evolved over the past twenty years in line with changes of government. In the 1980s, measures of performance centred on activity and cost and were eventually used for local management and central monitoring. From the beginning of the 1990s and until the change of government in 1997, indicators were still used to measure efficiency, but with a limited focus on public accountability.

In 1997, a new approach was adopted by the Labour Government, through the Performance Assessment Framework, which focused attention on the following areas⁸:

- Health improvement: reflecting the global aims of improving the general health of the population and reducing health inequalities, although these are influenced by many factors that reach well beyond the scope of the NHS.
- Fair access: recognising that the NHS must begin by offering fair access to health services and relate them to people's needs, irrespective of geography, socio-economic group, ethnicity, age or sex.
- Effective delivery of appropriate health care: recognising that fair access must involve providing effective, appropriate and timely care, and that this must comply with generally agreed standards.
- Efficiency: ensuring that effective care is delivered with a minimum of waste and that the NHS uses its resources to provide value for money.
- Patient/carer experience: assessing the way in which patients and their carers experience and view the quality of the care that they receive and give, and ensuring that the NHS is sensitive to individual needs.
- Health outcomes of NHS care: assessing the direct contribution of NHS care to improvements in overall health and completing the circle back to the global goal of health improvement.

Those indicators are updated on a regular basis (table 1). The Government has also developed indicators to rate the performance of NHS Trusts -acute, ambulance, mental health, and primary care- and specific programs through the Healthcare Commission. The ultimate aim is to identify and provide hospitals and other health care providers with appropriate incentives and management. These indicators are in line with government priorities with regards to NHS hospital performance goals.

This approach is one of the most complete to date. The framework covers all areas of the health system, with indicators reflecting the government priorities and particular public concerns. This kind of framework could be applied to other settings, such as cities, by adapting the indicators chosen so as to take into account specific concerns.

New Zealand

Although there is no single national framework for health performance indicators, an important part of policy development in New Zealand has centred on the development of indicators that focus on different aspects of the health care system, such as access to care, ethnic inequalities, and quality of care⁹. In 1996, the Ministry of Health entered into an annual funding agreement with the Health Funding Authority (HFA), which was maintained throughout the period during which the HFA was responsible for health service purchases in New Zealand. This funding agreement was the key document against which the HFA was monitored and assessed in terms of accountability. It set out key objectives and the measures and reporting requirements needed to monitor performance against these objectives. It also specified what funds would be made available to achieve these goals¹⁰. The agreement also outlined baseline services that the HFA was required to ensure and also terms of access to these services and safety standards.

While the objectives of both the health system and the government were related to priority outcomes, most of the performance measures for the HFA specified processes and outputs that could contribute to these outcomes. For example, in 1999/2000, 12 objectives were established, including: public guarantees concerning access, quality, and the security of services; timely, equitable and nationally consistent access to elective services; and reducing long-standing disparities in health status. Performance measures relevant to these objectives included:

- Improving public information relating to service coverage.
- Giving information to service providers.
- Achieving credible levels of access to surgical services.
- Purchasing services for Maori health priority areas.

Most of the performance targets set for the HFA appear to relate to process rather than output. However, data have been collected on all major issues, including waiting times for elective surgery, the health status of different socio-economic and ethnic groups, and service provision. Moreover, the Performance Management Unit at the Ministry of Health has prepared regular reports on such questions as risk-adjusted mortality and hospital readmission and complication rates in New Zealand.

The experience of New Zealand has been successful. The process of formalized contracting arrangements between the founder/government and a central purchaser appears to have improved accountability and transparency in care provision and purchasing in New Zealand by achieving better data collection and performance measures. By holding purchasers responsible for the delivery of outputs and processes while maintaining strict policy guidelines with respect to health targets and priority areas, it has been possible to allocate funds to areas that had previously been largely ignored. These include health care for the Maori population and the disabled and dental care. However, the impact of these arrangements on other health outcomes has been less clear. In particular, no evaluation seems to have been carried out on the performance of the HFA in comparison with the previous situation, in which four purchasing authorities and providers competed with each other.

Other experiences

Other performance assessment experiences are also worth mentioning, despite the fact that they either constitute only a partial assessment of the health care system or are currently at early stages in their development. One such case is the Dutch experience of measuring and improving the performance of home care services. This indicates that such processes can also be successfully applied in the area of long-term care¹¹.

Australia has several accreditation bodies, including the Australian Council on Healthcare Standards (ACHS), Medical Colleges, and several consortia of hospitals that are working to develop appropriate indicators and benchmarks and report on performance⁶. Since 2000, New Zealand has established balance score cards and has pooled information on costs, quality and outcomes that are now used to compare hospital performance and quarterly reports and are available to the general public¹¹.

Another example is the USA, whose many different performance measurement experiences have mainly been oriented towards guaranteeing the quality of the services provided by Health Maintenance Organizations^12,13. France has a national system for accreditation and evaluation¹¹. Sweden has national quality registers that have generated a large number of scientific publications and provided examples of improvements in the quality and cost of hospital care¹¹. Sweden has also played an active part in developing a common framework of performance indicators for the Nordic countries.

 

Discussion

The multiple experiences of performance assessment from around the world presented here bear testimony to the importance and usefulness of this topic for policy makers and populations alike. All the initiatives mentioned have been put into practice and are still being used and continuously improved. By drawing on trans-national experiences, countries may be able to adapt the frameworks developed by other states to meet their own particular circumstances^14-16. Individual health systems can learn the good practices and avoid the errors made elsewhere¹⁷. This kind of approach could also be used to assess health sector reforms with respect to system performance¹⁸.

The WHO World Health Report 2000 on performance assessment is the best known example of this type of analysis. Its importance derived from the relevance of the organization, the media impact of the report itself, and the debate that it subsequently generated. The report stimulated vigorous discussion of the processes associated with both its scientific content and ideological basis. The most important issues outlined included: the use of key informants, the definition of fairness, the statistical inference of missing data, the use of DALE instead of more traditional measures of health outcomes, and the use of a composite index^19,20. In this process, there were a number of methodological flows and discrepancies due to differences in ideological bases and the importance of the associated social consequences^21-24. It also had the effect of encouraging institutional critiques and initiatives. For example, the Pan American Health Organization was the subject of a series of meetings and discussions that culminated in a set of recommendations for performance assessment²⁵ and the OECD proposed a few modifications to the original WHO framework¹¹. One of the most positive contributions of academic exercises such as this is to present a framework for analysis and to identify a series of issues that are relevant for measuring health systems performance^1,26,27.

The OECD defines performance management as the whole set of institutional and incentive arrangements by which performance information is used to influence performance in health care systems¹¹. Performance assessment is only useful and important to the extent that it provides the evidence required for developing better policies, strategies and programs. In other words, the analytic component must go hand in hand with action in the field². The capacity to collect meaningful and consistent information on outcomes is of critical importance -particularly in relation to the means employed and the goals set-, because the availability or lack of availability of information about specific areas may reveal a lot about the strengths and weaknesses of a particular system.

For the assessment to be fruitful in a practical manner, a balance needs to be found between including all of the factors that might be interesting or relevant and affect outcomes and being able to produce timely information relating to them and to present this in a format that managers are able to digest¹. The indicators chosen need to be appropriate, relevant to both policy-makers and the general population, and sufficiently robust to allow measurement and respond to changes within the system. As such, they can be used to analyse progress over time. Any information should be presented in an understandable way, so that the stakeholders involved in the process -consumers, providers, managers, and policy makers- can satisfy their different needs.

At a later stage, wherever there is room for improvement, new strategies based on a careful diagnosis of the underlying causes of existing problems should be used to improve performance. The specific strategy (or strategies) chosen could vary from country to country or even within a given country, with targets depending on the characteristics and resources of the particular system and the associated professional and social values. These include: external oversight, knowledge/skill enhancement of providers, empowering consumers, incentives and regulation²⁸. They take different forms and often offer different prospects of success in accomplishing their objectives¹¹. The public disclosure of performance data could help to spark improvement, as it has in the clinical context^11,29.

Some interesting points can be highlighted from making comparisons between these different initiatives. Both the Australian and Canadian experiences tend to focus on assessing the health status of users and on health determinants. The Australian model has the advantage of facilitating reporting at both the national and local levels and for specific risk populations. In New Zealand, however, the main focus is on improving the flow of information to the public and to service providers. Special attention is also given to minority populations such as the Maori.

The work undertaken in the UK and Canada has been the most accomplished to date. These countries define precise categories, criteria and indicators to analyse and assess in their respective health care systems. Indeed, the use of categories allows a wider and more systematic vision of the health care system^30,31. This permits comparisons of the type undertaken in certain other countries^32-35 and includes the performance indicators that are also mentioned by the OECD^36,37. However, there are also differences, for while the Canadian approach is centred around health status, non-medical determinants of health and indicators of effectiveness, that of the UK tries to integrate certain aspects of health care that are more difficult to measure, such as responsiveness, patient and carer experience, and fairness. The UK also develops indicators relating to the specific health problems associated with urban contexts in greater detail.

In urban contexts, local governments tent to play an active role in policy development and assuring service delivery and public health functions³⁸. Performance assessment fits into these environments perfectly, since it is a useful tool for designing and monitoring the accomplishment of policies, and for assessing the performance of the services delivered³⁹. The framework should be applied to each particular context and take into account the specific characteristics of every city, including the composition of its population, environmental factors, and other local government policies.

Barcelona and Montreal are currently working on a collaborative project involving performance assessment in their respective health care systems. The indicators used in the UK initiative are probably the most suitable ones to apply in these two cities, as they take into account general dimensions of health and health care, while also focusing on areas that are more specific to the particular health problems of a urban setting (e.g. AIDS, the health of immigrants, concerns about community care, particular types of cancer and geographic inequities). This is, for example, the case of monitoring HIV/AIDS rates. In urban contexts, this is of particular importance, as these tend to be the settings most exposed to the propagation of epidemics. The approach can be used as an indicator of the access that different socio-economic groups have to risk disease prevention and health promotion services. It also serves as an indicator of health outcomes, because it can show how effective public health services are at reducing risk and risk factors. Another indicator is the rate of hospitalisation on the grounds of drug abuse, which is relevant for cities confronted with the problem of drug abuse. This could also be used to determine access to risk prevention and health promotion services, and the equity of access to these services. Another interesting indicator could be the rate of mental illness and the incidence of premature deaths, which are both relevant in urban contexts.

Basing their model on the UK indicators, the next step in the collaboration between Barcelona and Montreal will be to develop a set of indicators that could be applied in both cities. Some preliminary results are presented in table 2. We are currently working to produce a whole set of indicators for each of the dimensions proposed, which will include specific definitions and guidelines for data collection. We hope to soon have our first results and to then be able to judge their applicability in practical terms.

 

Acknowledgements

We are grateful to the Consorci Sanitari de Barcelona, Corporació Sanitària de Barcelona, and Agence de développement de réseaux locaux de services de santé et de services sociaux de Québec, who are our partners in this project.

The members of the Barcelona and Montreal group on the performance of health care services are: Josep Benet, Mike Benigeri, Carme Borrell, Marc Bourguignon, Xavier Castells, Louis Côté, Danièle Dorval, Jaume Estany, Gonçal Foz, Lluís Franch, Anna García-Altés, Josep M. Giralt, Àlex Guarga, José Manuel Pérez-Castejón, Antoni Plasència, Alfonso Pozuelo, Aina Plaza, Elvira Torné, Corinne Zara, and Lauriane Zonco.

 

Bibliografía

1. Pan American Health Organization. Health system performance assessment and improvement in the region of the Americas. Washington: PAHO; 2001.        [ Links ]

2. Hurst J. Performance measurement and performance management in OECD health systems: an overview of issues and challenges. «Measuring up» Improving health systems performance in OECD countries. OECD health conference on performance measurement and reporting. Ottawa, November 2001.        [ Links ]

3. Murray CJL, Frenk J. A framework for assessing the performance of health systems. Bull World Health Organ. 2000;78: 717-31.        [ Links ]

4. Borrell C, Pasarín M. Desigualdad en salud y territorio urbano. Gac Sanit. 2004;18:1-4.        [ Links ]

5. World Health Organization. The World Health Report 2000: Health systems performance. Geneva: World Health Organization; 2000.        [ Links ]

6. National Health Performance Committee. Fourth national report on health sector performance indicators. A report to the Australian Health Ministers' conference. Sydney: NSW Health Department; 2000.        [ Links ]

7. Canadian Institute for Health Information. Health indicators. Ottawa: Canadian Institute for Health Information; 2003. Catalogue n.o 82-221-XIE, Volume 2003, N.o 2.        [ Links ]

8. National Health Service. NHS Performance Indicators. National Figures: February 2002. London: National Health Service; 2003.        [ Links ]

9. Or Z. Improving the performance of health care systems: from measures to action (a review of experiences in four OECD countries). Labour market and social policy. Occasional papers n.o 57. Paris: OECD; 2002.        [ Links ]

10. New Zealand Ministry of Health. Health Funding Authority Performance Report. Quarter Four 1999/2000. Wellington: New Zealand Ministry of Health; 2000.        [ Links ]

11. Hurst J, Jee-Hugues M. Performance measurement and performance management in OECD Health Systems. Labour Market and Social Policy, Occasional Papers N.o 47. Paris: OECD; 2000.        [ Links ]

12. McIntyre D, Rogers L, Heiner EJ. Overview, history and objectives of performance measurement. Health Care Financing Review. 2001;22:7-21.        [ Links ]

13. Feachem RGA, Sekhri NK, White KL. Getting more for their dollar: a comparison of the NHS with California's Kaiser Permanente. BMJ. 2002;324:135-41.        [ Links ]

14. Evans DB, Edejer TT, Lauer J, Frenk J, Murray CJ. Measuring quality: from the system to the provider. Int J Qual Health Care. 2001;13:439-46.        [ Links ]

15. Upshaw V. The National Public Health Performance Standards Program: would it strengthen governance of local public health? J Public Health Pract. 2000;6:1-18.        [ Links ]

16. Mc Pake B, Mills A. What can we learn from international comparisons of health systems and health system reform? Bull World Health Organ. 2000;78:811-20.        [ Links ]

17. Mannion R, Goddard M. Performance measurement and improvement in health care. Appl Health Econ Health Policy. 2002;1:13-23.        [ Links ]

18. Knowles JC, Leighton C, Stinson W. Measuring results of health sector reform for system performance: a handbook of indicators. Special Initiatives Report N.o 1. Bethesda: Partnerships for Health Reform, Abt Associates Inc.; 1997.        [ Links ]

19. Coyne JS, Hilsenrath P. The World Health Report 2000: Can health care systems be compared using a single measure of performance? Am J Pub Health. 2002;92:30-3.        [ Links ]

20. Appleby J, Street A. Health system goals: life, death and... football. J Health Serv Res Policy. 2001;6:220-5.        [ Links ]

21. Almeida C, Braveman P, Gold MR, Szwarcwald CL, Mendes Ribeiro J, Miglionico A, et al. Methodological concerns and recommendations on policy consequences of the World Health Report 2000. Lancet. 2001;357:1692-7.        [ Links ]

22. Navarro V. Assessment of the World Health Report 2000. Lancet. 2000;356:1598-601.        [ Links ]

23. Fernández-Díaz JM. El informe de la Organización Mundial de la Salud 2000: un juego de ordenador políticamente incorrecto. Gac Sanit. 2001;5:453-60.        [ Links ]

24. Murray CJL, Salomon JA, Mathers C. A critical examination of summary measures of population health. Bull World Health Organ. 2000;78:981-94.        [ Links ]

25. Pan American Health Organization. Final Report on regional consultation of the Americas on health systems performance assessment. Washington: PAHO; 2001.        [ Links ]

26. Handler A, Issel M, Turnock B. A conceptual framework to measure performance of the public health system. Am J Public Health. 2001;91:1235-9.        [ Links ]

27. Beaulieu J, Scutchfield FD. Assessment of validity of the national public health performance standards: the local public health performance assessment instrument. Public Health Reports. 2002;117:28-36.        [ Links ]

28. Leatherman S. Applying performance indicators to health system improvement. OECD/Canada Conference. Measuring up: improving health systems performance in OECD countries. Ottawa, November 2001.        [ Links ]

29. Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data. What do we expect to gain? A review of the evidence. JAMA. 2000;283:1866-74.        [ Links ]

30. DeRosario JM. Healthcare system performance indicators: a new beginning for a reformed Canadian healthcare system. J Healthc Qual. 1999;21:37-41.        [ Links ]

31. Handler A, Issel M, Turnock B. A conceptual framework to measure performance of the public health system. Am J Public Health. 2001;91:1235-9.        [ Links ]

32. Evans DB, Tandon A, Murray CJ, Lauer JA. Comparative efficiency of national health systems: cross national econometric analysis. BMJ. 2001;323:307-10.        [ Links ]

33. The Conference Board of Canada. Challenging Health Care System Sustainability: Understanding Health System Performance of Leading Countries. Ottawa: the Conference Board of Canada; 2004.        [ Links ]

34. Hussey PS, Anderson GF, Osborn R, Feek C, McLaughlin V, Millar J, Epstein A. How does the quality of care compare in five countries? Health Aff (Millwood). 2004;23:89-99.        [ Links ]

35. OECD. Towards High-Performing Health Systems. Paris: OECD; 2004.        [ Links ]

36. Jee M, Or Z. Health outcomes in OECD countries: a framework of health indicators for outcome-oriented policymaking. Labour market and social policy. Occasional papers n.o 36. Paris: OECD; 1999.        [ Links ]

37. Marshall M, Leatherman S, Mattke S, and the Members of the OECD Health Promotion, Prevention and Primary Care Panel. Selecting indicators of for the quality of health promotion, prevention and primary care at the health system level in OECD countries. OECD Health Technical Papers n.o 16. Paris: OECD; 2004.        [ Links ]

38. Líndez P, Villalbí JR, Vaqué J. Funciones, actividades y estructuras de salud pública: el papel de los municipios grandes y medianos. Gac Sanit. 2001;2:164-71.        [ Links ]

39. Derose SF, Schuster MA, Fielding JE, Asch SM. Public health quality measurement: concepts and challenges. Annu Rev Public Health. 2002;23:1-21.        [ Links ]

 

 

Dirección para correspondencia:
Dra. Anna García-Altés.
Agència de Salut Pública de Barcelona.
Pl. Lesseps, 1. 08023 Barcelona. Spain.
E-mail: agarcia@aspb.es

Recibido: 25 de julio de 2005.
Aceptado: 21 de febrero de 2006.