Monitoring the quality of an Italian public psychiatric service:
A four dimensions study of the outcome

Antonio Amatulli* Paolo Azzone*,** Debra Srebnik*** * Ospedale G. Salvini, Garbagnate Milanese - Milan, Italy ** Scuola di Specializzazione in Psichiatria, Università degli Studi di Milano - Milan, Italy. *** Department of Psychiatry and Behavioral Sciences - University of Washington, Washington, D.C., USA

Key words: Quality assurance, Psychiatric services, Satisfaction, Functioning, Quality of life, Clinical status.

---

ABSTRACT - Objective: Within the concept of the outcome of hea1th services, the user's satisfaction has to be integrated by other more objective measures of health and quality of life. Debra Srebnik and coworkers have proposed a Survey for Monitoring the Quality of Public Mental Hea1th Services (SMQPMHS), which covers the following dimensions: Satisfaction, Functioning, Quality of Life and Clinica1 Status. The research main goals were as follows: a) to study the psychometric properties of the Ita1ian version of the Survey, and b) to study possible differences between the outcome of psychiatric patients applying to our outpatient facilities vs. the American sample.
Methods: The Italian version of SMQPMHS was proposed to all patients receiving care at 2 Italian outpatient psychiatric facilities over a period of 2 months. 291 subjects accepted to participate in the study and fi1led in adequately the questionnaire.
Results: Mean scores of the 13 variables of the Survey were very similar to those observed in the Arnerican sample. A matrix of correlations between each variable and each of the others indicated adequate internal consistency. A principal component analysis supported the four-dimensions model of the Survey. Patients recently hospitalized showed a poorer Clinical Status and a lower Satisfaction. Patients unemployed at the time of the survey showed a poorer Clinical Status.
Conclusions: The Italian version of SMQPMHS showed good psychometric properties, even though concurrent validity needs further study. No differences emerged between the present sample and the American sample.

---

 

Introduction

Measurement and evaluation of what mental health services do for people is more and more important at all levels of clinical practice (Goldman et al. 1990). Administrators demand performance reports for their system of budget management; service providers can no longer consider themselves being (and being worth) only what they do; consumers are growing more and more aware of having the right to rate the care they receive on its merits; and the members of the families of the patients act as custodians or guarantors of all these aspects. Too often, health care professionals have failed to seek the participation of the clients in choices that are inherent to their care, and this often means that they made decisions "in the patient's exclusive interest" without really knowing what that interest is! (Wallace et al. 1999).

From the point of view of the users of the mental health service, obtaining their perspective on care quality contributes to their sense of freedom of choice, autonomy and empowerment. Therefore growing emphasis is being placed in mental health care, on the use of the perceptions and degree of satisfaction of the users in the evaluation of the quality of the care provided (Holcomb et al. 1998) (Davis & Fong 1996, Ruggeri et al. 2000, Parasuraman et al. 1988, Rosenthal & Shannon 1997). Following these precepts, many instruments have been developed and used, both for purposes of research and for the purpose of reorganizing service processes.

It is a well-known experience that questionnaires specifically designed for the rating of satisfaction tend to receive a generally positive range of responses, thus there is still much work to be done, at the level of developing methods and procedures for patients to rate quality of the care and assistance provided (Cochrane 1972, Zastovny et al. 1995).

One of the dimensions most commonly taken into account in evaluation of the outcome and quality of treatments is the degree of consumer satisfaction (Khayat & Salter 1994, Aharony & Strasser 1993, Ross et al. 1995, Sitzia & Wood 1997). In Italy even the national and regional health plans emphasize the consumer's satisfaction as a key principle orienting service organization and the selection of treatments which should be made available to the patients.

While a user-oriented service does need management, the assessment of the perception and the degree of the patient's satisfaction also involves challenging clinical and methodological issues (Cleary et al. 1991). For instance, Risser has identified certain dimensional aspects of user satisfaction, such as costs, the personal qualities of the treating physician and the nature of the interpersonal relationship, the professional skill of the physician and the perceived quality of the treatment received (Risser 1975). Ware et al. (1983) pointed out other related factors [e.g., interpersonal relations (respect, courtesy, sympathy), technical aspects of the treatment (its appropriateness and accuracy), accessibility of treatment (logistics, waiting periods), financial aspects, effectiveness and outcome, continuity of treatment and others].

It is therefore quite clear that whatever means are used to investigate the perception and degree of satisfaction of the user, it is essential to assess all the areas involved.

Finally, it is important to determine how the services can make use of the patients' perceptions to improve the effectiveness of treatment and the quality of the care processes. Patient perspectives can be used in reviewing or verifying and reengineering both organization and management (Cleary & Edgman-Levitan 1997, Barnes et al. 1999). Thus it is to be hoped that health care will proceed to operate with a view to continuous improvement of the quality of service, based on periodic multidimensional assessments.

Assessments of the perceptions and/or degree of satisfaction of the patients (for example in a mental health center) should include an expanded view of the functions of the person and the sphere in which he lives. For example, the conceptual model relative to the outcomes of psychiatric patients proposed by Rosemblatt and Attkisson may be useful in overcoming a one-dimensional reading of the performance of the user. These authors have proposed four areas of evaluation of the outcome: psychopathological and symptomatological, personal and social functioning, the degree of satisfaction and the quality of life (Rosemblatt & Attkisson 1993). This means that appropriate predictors have to intercept and correlate with these areas. This model was used by Srebnik and her colleagues to develop outcome indicators that would "draw" in a sufficiently complete manner the dimensions of quality relative to a mental health service (Srebnik et al. 1997).

We decided to repeat the study in Italy. We first translated the instrument to Italian, then back-translated it to English, and the authors reviewed it for accuracy.

The main goals of the present study can be summarized as follows:

1. To assess whether the psychometric properties of the English version, and particularly the internal consistency and the underlying factor structure had been preserved in the Italian one.

2. To compare satisfaction data from the study of Srebnik et al. of users of Mental Health Centers (MHCs) in the United States with data from users of Italian MHCs. In fact, both settings studied offered mental health service on an outpatient basis, but a number of peculiarities of the Italian psychiatric system might have been the basis for a greater or lower satisfaction: a) a wider role of psychiatrists in the Italian MHC, including the functions covered by non clinical case managers in the US psychiatric facilities, b) an extremely rigid job market c) a very limited availability of sheltered houses, and d) the specific cultural framework the patients were living in, possibly determining different social tolerance towards the expression of hostility or insatisfaction towards health institutions.

Methods

Setting

The study was carried out in two outpatient psychiatric structures in the Psychiatric Unit of Garbagnate Milanese, classified one as a MHC and the other as a Psychiatric Outpatient facility. Both structures offered, at the time of collection of the data, health and psycho-social services to patients affected by mental illness, resident in seven municipalities in the north of the province of Milan. The area of our Health Care Unit included both suburban and rural zones.

Together, the two centers served 1836 people. The services provided were as follows: psychiatric visits, psychological interviews, psychotherapy, psychological tests, house visits, nursing activities in the centers or in the patients' homes, psycho-social activities for the purpose of helping the patients to re-socialize or work, and consultation by social workers.

Measurements

The Italian translation (made by the authors) of the Survey for Monitoring the Quality of Public Mental Health Services (SMQPMH) consists of a self-administered questionnaire that includes a total of 45 items, from which 12 measurements are drawn, organized into 3 domains or areas, supplemented by a case manager rated Clinical Status scale. The following 4 domains are therefore covered by the client self-report and clinician-rated instruments: Satisfaction, Functioning, Quality of Life and Clinical Status.

The domain of Satisfaction measures the degree of satisfaction of the users with their involvement in the therapeutic programme, with the appropriateness of the therapy and security relative to their visits to the mental health center. The Functioning area measures the physical, mental and social functioning of the patients and their ability to manage their lives and their symptoms. The domain of the Quality of Life measures safety in the home, achievement of goals and experiences of violence. Finally, Clinical Status, rated by clinicians, is measured by symptoms, functioning, substance abuse condition and compliance with treatment.

The survey was developed to measure the outcome of psychiatric treatment. It was drawn up in such a way as to include the aspects of outcome that are most important for the different stakeholders of psychiatric services, such as users, members of their families, professionals and administrators. The questionnaire was drawn up making ample use of existing instruments designed to measure separate dimensions of the outcome of psychiatric treatments. Specifically, the survey included the eight-item Client Satisfaction Questionnaire (CSQ, Nguyen et al. 1983), the SF-12 (Bogaert-Martínez et al. 1996, McHorney et al. 1993), seven items from the Lehman Quality of Life Interview (Lehman 1991), and four items from a California Public Mental Health Survey (Veit 1995). The other items were developed by Srebnik and coworkers. The case manager survey consisted of the Four-Dimensional Classification Scale (Comtois et al. 1994), which includes four single-item 7-point scales that assess symptoms, functioning, substance abuse and treatment compliance. The original study, carried out in the United States on 236 users of outpatient mental health services, showed that the SMQPMH scales representing each dimension showed adequate internal consistency and evidence of concurrent validity. Furthermore, the analysis of the main components confirmed the 4 dimensional model that underlies the instrument.

Subjects

The questionnaire was submitted to all the users who had consecutively received health care or psychosocial services from the two sites of the study during a period of two months (June-July 2000), provided that was not their first contact with the centers. Out of 317 patients receiving the questionnaire, 309 agreed to fill it out and return it. Filling it out took between 10 and 25 minutes; the personnel at the Center were available to provide a little help when necessary. At the same time, the treating physician filled out the Clinical Status component. After collecting the questionnaires, we added the personal data and the number of specialized contacts the patients had received in the six previous months.

Since some of the questionnaires were only partly completed, we included in the study only those questionnaires that contained enough information to establish at least 8 measurements out of 13.

Of the 291 users who satisfied this condition about half were women and about half (46%) were married. The average age was 43.35 years (SD = 15.1). Fewer than 20% of the patients had finished high school and fewer than 2% had graduated from university. The rather low social and cultural level of the sampling was confirmed by the very limited number of professionals and self-employed. Among other things, more than a third of the sample did not perform any productive activity.

The diagnoses were mainly Affective Psychosis (31.7%) and Schizophrenic Disorders (30.6%), but the sample also included patients with Neuroses (17.0%) or Personality Disorders (11.4%). Compared to the prevalence of the various diagnoses in the general population of patients of our Service, significant differences were observed only for Schizophrenic Disorders and Organic Mental Disorders. Specifically, our sample showed a higher prevalence of Schizophrenic Disorders (30.63% vs. 21.02%, X² = 12.59, p = 0.0004) and a lower prevalence of Organic Disorders (3.32% vs. 8.99%, X² = 10.01, p = 0.0016).

Data analysis

The internal consistency of the Italian version of the questionnaire was assessed through a matrix of correlations between each of the measurements of outcome and each of the others, both those belonging to the same domain and those belonging to different domains. The assignment of several measurements to the same domain in the survey implies the assumption that these measurements represent underlying aspects that are analogous and interdependent. We expected, therefore, that the correlations among measurements belonging to the same domain would be stronger than those observed among different domains.

In order to determine whether the underlying factorial structure of the survey remained stable in the Italian version as well, we also performed a principal components analysis. The reliability of each measurement of the scale was examined using Cronbach's alpha coefficient. Lastly, the concurrent validity of the values of the 4 areas of the survey was verified by analysis of variance, using as factors of classification the state of employment and the existence of episodes of hospitalization in specialized environments.

Results

Average values of the Questionnaire in the population investigated

Table I presents the average values of each measurement of the sampling. A comparison with the values obtained by Srebnik and her collaborators applying the SMQPMH to the users of a number of territorial psychiatric services in the U.S. shows considerable convergence between the two populations. However, the values relative to physical and mental functioning cannot be directly compared, as the American authors report values obtained after standardization of the data.

Internal consistency of the scale

Table II shows the correlations between each of the measurements of outcome and each of the remaining measurements belonging both to the same domain and to different domains. All the reciprocal correlations between measurements of the domain of Satisfaction and measurements of the domain of Functioning are significant with a median of correlation coefficients of 0.49 (range 0.34 - 0.71 vs. range 0.24 - 0.58 in the American sample) for the domain of Satisfaction and 0.55 (range 0.34 - 0.71 vs. range 0.17 - 0.60 in the American sample) for that of Functioning. Among measurements belonging to the area of the Quality of Life, the correlations observed were not as strong, with only three statistically significant values out of six, with a median of correlation coefficient of 0.14 (range -0.01 - 0.34 vs. range -0.08 - 0.27 in the American sample). That might be due to a "floor" effect, as criminal offences are relatively rare and consequently most patients had no victimization or safety issue.

On the whole, the correlation among measurements belonging to different domains appeared less strong than among measurements belonging to the same domain. The measurements of Functioning and Satisfaction showed a number of significant correlations, but the association appeared less strong than that which was observed within each of the two domains, with a median correlation coefficient of 0.23 (range 0.09 - 0.37 vs. range: -0.05 - 0.21 in the American sample). Half of the correlations between measurements of the domain of the Quality of Life and of Functioning were significant (range -0.11 - 0.54 vs. -0.11 - 0.30 in the American sample), while the relationship with the domain of Satisfaction appeared slight (range 0.02 - 0.31 vs. -0.11 - 0.30 in the American sample).

On the whole, the results of the matrix of correlations seem to confirm the structure of the questionnaire and are in line with the data reported by the American study of validation of the instrument. With respect to that study, it is possible to observe correlations tending on the average to be stronger between the area of Clinical Status and the other areas (ranges: 0.12 - 0.25, 0.23 - 0.35, and 0.12 - 0.19 for Satisfaction, Functioning and Quality of Life respectively; vs. 0.13 - 0.30, -0.05 - 0.06 and 0.10 - 0.14 in the American sample). On the other hand, the values of r observed testify to the existence of a significant correlation between Clinical Status and the other components of outcome, but indicate also that Clinical Status explains only a limited part of the variability observed.

It seems to us that these values might be explained in part by the higher level of competence of the clinicians in our survey, all of them being physicians specialized in Psychiatry, as compared with the context of the American survey, in which bachelor level case-managers most often completed the Clinical Status Component.

Factor analysis

The results of the principal components analysis are shown in Table III.

 

In order to check the structure of the instrument within the sample used for the study, we selected a solution with 4 factors. Each factor showed eigenvalues higher than 1. Assigning each measurement to the factor for which the measurement presented the highest factor loading value, the 4 measurements of the domain of Functioning could be assigned to factor 1, the measurements of Satisfaction to factor 2 and the Clinical Status to factor 4. The measurements of the area of the Quality of Life exhibited a more complex behavior. The measurements of concerns about Living conditions and Victimization presented higher factor loadings on factor 3, while Safety and Goals Attainment presented higher factor loadings on factors 1 and 4 respectively. Similar problems were also found for the factor relative to the Quality of Life in the study by Srebnik and collaborators. We agree with the American authors that reasons of clarity of interpretation suggest the inclusion of these last two measurements in the Quality of Life domain.

Global outcome indicators

The Clinical Status component yielded a single global score representing the whole domain of mental illness. Global indicators of Satisfaction, Quality of Life and Functioning were obtained by taking the mean of the different measurements of each domain, after the separate measures have been standardized around a mean of 10 and a standard deviation of 1. For the domain of Satisfaction the mean thus obtained was 10.03 ± 0.77. The internal reliability, measured using Cronbach's alpha coefficient, was 0.76. For the domain of Functioning, the global outcome indicator was found to be 10.01 ± 0.83 with a Cronbach's alpha coefficient of 0.56. On the average, the patients included in the sampling presented a global indicator of the Quality of Life of 10.05 ± 0.57 with a Cronbach's alpha coefficient of 0.27.

Measurements of concurrent validity

As was done in the study by Srebnik and collaborators, the validity of the SMQPMH was tested in terms of concurrent validity. Thus, we studied the relationship among the values of the global outcome indicators emerging from the survey and two measures representative of dimensions commonly believed to be associated with a good outcome of mental health treatment. The dimensions considered were the absence of hospitalization for psychiatric problems during the study period and occupation in a productive job.

As regards hospitalization for psychiatric problems, we compared the averages of the global indicators of outcome of patients hospitalized during the study period with the averages of patients who had not been hospitalized during the same period. We expected that patients who had been hospitalized would present a generally poorer outcome with respect to at least a few indicators. The analysis of variance (Table IV) showed a significant difference for two dimensions: Satisfaction and Clinical Status. In both cases, the patients who had been hospitalized during the study period presented outcomes that were worse than those of the patients who were not hospitalized.

With respect to the condition of employment, we compared the outcome indicators of unemployed patients or invalids with those of employed or retired patients. We expected the outcome indicators for patients who had performed productive work to be better than those of patients who had not been able to work during the study period. This difference was effectively observed for only one of the outcome indicators, Clinical Status. As was foreseeable, employed patients had a better Clinical Status than unemployed ones, and the difference was statistically highly significant (Table V).

On the whole, these results confirm the concurrent validity of the Clinical Status component and in part of the Satisfaction global outcome indicator. We think these data indicate that a more comprehensive concurrent measurement of outcome is necessary for a better assessment of the validity of the survey as a whole in our work setting.

Outcome of patients in relation to mental health services and the services they provide

In order to study the effect of the services provided by our mental health structures on the outcome of the patients of those structures, we examined the relationships between the outcome indicators provided by the SMQPMH and the interventions effectively received by the users. We took into consideration 4 types of services: psychiatric visits, rehabilitation in day care facilities, nursing care and the actions of social workers. Under the heading of psychiatric visits we considered the number of contacts received in the six months prior to collection of the data. On the average, the patients who participated in the study had been seen 4.09 ± 3.69 times by a specialist during the period reviewed. As regards nursing care, rehabilitation in day care facilities and actions by social workers, we divided the patients into two groups, depending on whether or not they had received one or more of these services during the period of the study.

Since the attention and the activity of psychiatric services tend to concentrate on patients with more severe symptoms in psychopathological terms, we expected that the patients who had received more services would be characterized by a worse Clinical Status. We expected, however, that the patients who had been seen by a specialist more times or had received a broader range of services would present a better outcome, or at least a higher level of Satisfaction.

The correlations between the number of psychiatric visits received and each of the global outcome indicators did not give statistically significant results. The results of the analysis of variances confirmed that the patients who received a broader range of services had a worse Clinical Status (F = 7.75, df = 1,269, p = 0.0058 for social worker's interventions; and F = 15.52, df = 1,269; p = 0.001 for nursing care), but did not show any effect of these actions on the other outcome indicators, particularly on Satisfaction. Indeed, the patients who received more attention from the social worker or more day care rehabilitation showed a lower level of Satisfaction than the other patients (F = 24.52, df = l,23l, p = 0.00000l for social worker's interventions; and F = 6.39, df = l,23l, p = 0.012 for day care rehabilitation). This could perhaps be explained by the discrete level of correlation observed between the measurements of the domain of Satisfaction and those of the domain of Clinical Status. Since in our sample, as we have already pointed out, the more seriously ill patients tend to be less satisfied, it is likely that the subgroup of patients who received social and rehabilitation services, being characterized by a more serious Clinical Status, would tend to perceive the cares received in a more negative manner.

Discussion

The Italian version of the SMQPMH has proved to be an easily administered, practical instrument for studying different aspects of the outcomes of patients treated by our outpatient services. Its psychometric properties have been found to be globally conserved in spite of its ease of use and simplicity. Furthermore, administering it in our service gave results that are for the most part comparable with those of the original American study, thus confirming its reliability.

To the basic format, we then added indicators of utilization of the service in order to further investigate the effect of the psychiatric help performed by our mental health structures on the outcome of the patients. For this purpose, we took into consideration 4 types of interventions: psychiatric and psychological visits, rehabilitation in day care facilities, nursing care and the actions of a social worker. The results of this further investigation showed contrasting results that will have to be confirmed by a future repetition of the study, which we have planned.

It is important, however, to report that, although it appeared plausible to us to expect a higher level of Satisfaction on the part of patients who had received more care, we did not find this to be the case. In actual fact, these first results would appear to indicate that a poorer Clinical Status tends to characterize users who are subjected to a wider range of care, and for this reason their level of Satisfaction is lower. This seems to create an intrinsic limit to the studies on user satisfaction: either they will have to be geared to a less severely ill segment of the population (which is in any case numerically predominant) that uses the services, or they are likely to give very misleading results, exactly as we have just said. In addition, scores that are in any case "flattened" upward could lead to the dangerous and groundless inference that the cares provided are all of good or high quality.

As evaluations of this kind are rather common, it was necessary to decide how to deal with this type of difficulty of assessment. For example, the authors of the SMQPMH themselves recently described procedures of "risk adjustment", designed to "weigh" the scores achieved against the severity of the symptoms shown (Hendryx et al. 1999). Similarly, one could also simply group scores by clinical severity level. This is an avenue of study and research in the field of the evaluation of perceptions and user satisfaction that we consider of great importance in terms of method.

In relation also to these problems, the SMQPMH certainly had advantages: they include the multidimensional aspect of evaluation, in which the values of satisfaction are expressed with the other aspects of the outcome. The further quantification that we added regarding the extent of utilization of the service resources by each patient provides additional help in this direction. Some authors, however, point to other limits that must be understood and possibly dealt with when studies on perceptions and outcomes of treatments are used in the evaluation of the processes or quality of the cares provided. In fact, these studies fail to intercept an important (and often severely ill) group of patients: persons who suffer from psychiatric problems but do not have access to the service. Or all those individuals who leave the out or inpatient treatment circuit prematurely (Stockdill 1992).

In the last analysis, we can point to a number of factors that make the SMQPMH and its Italian translation an extremely useful instrument in the assessment of the patients' perceptions and of additional outcome indicators.

First, there is its extremely manageable aspect (the small number of items -only 45 in all for the four parameters- that differentiate it from most of the existing questionnaires) and its ease-to-use.

Second, there is the possibility that it can be used for more general evaluation of different aspects of outcome, as well as for identifying "critical areas" of the processes of care and treatment, which can then be modified.

Finally, comparisons can be made between data drawn in different services (with the precaution of the variable "weight" of the prevalence of different classes of disorder). The SMQPMH, which in any case has shown, in the Italian translation, results that are in most part comparable with those of the original American version, can prove to be an extremely useful instrument with a view to continuous quality improvement.

Its use is simple and inexpensive, repetition of data collection is therefore not too complex and even just by comparing the average values of the scales some types of evaluation (changes, critical aspects) appear extremely simple. Repetition of data collection in time will make it possible to monitor numerous outcome indicators (but also indicators of processes and structures) that lie at the base of a system for monitoring the quality of a community based psychiatric service.

Finally, our service has just undertaken a second data collection programme using the survey again. There are also plans for further studies of outcome validation in other parts of Italy.

References

Aharony L, Strasser S. Patient satisfaction: What we know about and what we still need to explore. Med Care Rev 1993; 50(1): 49-79.         [ Links ]

Barnes J, Stein A, Rosemberg W. Evidence-based medicine and evaluation of mental health services: Methodological issues and future directions. Arch Dis Child 1999; 80(3): 280-285.         [ Links ]

Bogaert-Martinez E, Caen E, Wilson W, et al. The SF-36 as a measure of functioning and health-related quality of life in individuals with severe and persistent mental illness: psychometric properties and normative data. Paper presented at the Annual Conference on State Mental Health Agency Services Research and Program Evaluation, Alexandria, VA, 1996.         [ Links ]

Cleary PD, Levitan SE, Roberts M et al. Datawatch. Patients evaluate their hospital care: a national survey. Health Aff 1991; 10(4): 254-267.         [ Links ]

Cleary PD, Edgman-Levitan S. Health care quality. Incorporating consumer perspectives. J Am Med Assoc 1997; 278(19): 1608-1612.         [ Links ]

Cochrane AL. Effectiveness and efficiency. London: Nuffield Provincial Hospitals Trust, 1972.         [ Links ]

Comtois KD, Ries R, Armstrong HE. Case manager ratings of the clinical status of dually diagnosed outpatients. Hosp Community Psychiatry 1994; 45: 568-573.         [ Links ]

Davis D, Fong M. Measuring outcomes in psychiatry: an inpatient model. Jt Comm J Qual Improv 1996; 22(2): 125-133.         [ Links ]

Goldman L, Francis Cook E, Oran J et al. Research training in clinical effectiveness: Replacing "in my experience..." with rigorous clinical investigation. Clin Res 1990; 38(4): 686-693.         [ Links ]

Hendryx MS, Dyck DG, Srebnik D. Risk-adjusted outcome models for public mental health outpatient programs. Health Serv Res 1999; 34(1Pt 1): 171-195.         [ Links ]

Holcomb WR, Bertman BD, Hernme CA et al. Use of a new outcome scale to determine best practices. Psychiatr Serv 1998; 49(5): 583-585, 595.         [ Links ]

Khayat K, Salter B. Patient satisfaction surveys as a market research tool for general practices. Br J Gen Pract 1994; 44(382): 215-219.         [ Links ]

Lehrnan AF. Quality of Life Interview Core Version. Baltimore: University of Mary land, Center for Mental Health Services Research, 1991.         [ Links ]

McHorney C, Ware J, Raczek A. The MOS 36-item SF-36: II. Psychometric and clinical tests of validity and measuring physical and mental health constructs. Med Care 1993; 31: 247-263.         [ Links ]

Nguyen T, Attkisson C, Stegner B. Assessment of patient satisfaction: development and refinement of a service evaluation questionnaire. Eval Program Plann 1983; 6: 299-314.         [ Links ]

Parasuraman A, Zethrnal VA, Barry LC. SERVQUAL: A multiple-item scale for measuring consumer perceptions of service quality. J Retailing 1988; 64: 12-40.         [ Links ]

Risser N. Development of an instrument to measure patient satisfaction with nurses and nursing care in primary care settings. Nursing Research 1975; 24(1): 45-52.         [ Links ]

Rosemblatt A, Attkisson CC. Assessing outcomes for sufferers of severe mental disorder: A conceptual framework and review. Eval Program Plann 1993; 16: 347.         [ Links ]

Rosenthal GE, Shannon SE. The use of patient perceptions in the evaluation of health care delivery systems. Med Care 1997; 35(11 Suppl.): NS58-68.         [ Links ]

Ross CK, Steward CA, Sinacore JM. A comparative study of seven measures of patient satisfaction. Med Care 1995; 33(4): 392-406.         [ Links ]

Ruggeri M, Lasalvia A, Dell' Agnola R et al. Development, internal consistency and reliability of the Verona Service Satisfaction Scale-European Version. Br J Psychiatry 2000; 177(suppl. 39): 41-48.         [ Links ]

Sitzia J, Wood N. Patient satisfaction: a review of issues and concepts. Soc Sci Med 1997; 45(12): 1829-1843.         [ Links ]

Slade M, Thornicroft J, Glover G. The feasibility of routine outcome measures in mental health. Soc Psychiatry Psychiatr Epidemiol 1999; 34(5): 243-249.         [ Links ]

Srebnik D, Hendryx M et al. Development of outcome indicators for monitoring the quality of public mental health care. Psychiatr Serv 1997; 48(7): 903-909.         [ Links ]

Stockdill J. A government manager's view of mental health advocacy groups. Adm Policy Ment Health 1992; 20: 45-55.         [ Links ]

Thornicroft G, Tansella M. Translating ethical principles into outcome measures for mental health services research. Psychol Med 1999; 29: 761-767.         [ Links ]

Veit S. California Mental Health Performance Outcome Project. Report for the California State Department of Mental Health. Sacramento, CA: Department of Mental Health, 1995.         [ Links ]

Wallace T, Robertson E, Millar et al. Perceptions of care and services by the clients and families: a personal experience. J Adv Nurs 1999; 29(5):1144-1153.         [ Links ]

Ware JE, Snyder MK, Wright R et al. Defining and measuring patient satisfaction with medical care. Eval Program Plann 1983; 6(3-4): 247-263.         [ Links ]

Williams B. Patient satisfaction: A valid concept? Soc Sci Med 1994; 38(4): 509-516.         [ Links ]

Zastovny TR, Stratmann WC, Adams EH et al. Patient satisfaction and experience with health services and quality of care. Qual Manag Health Care 1995; 3(3): 50-61.         [ Links ]

 

Address of correspondence:
Dott. Paolo Azzone
Via San carlo 131/b
20017- Rho (MI)
Fax. +39029309903
ITALY
e-mail: paoloazzone@hotmail.com