Introduction

There is growing interest in studying the quality and efficacy of mental health services¹. Perceived quality in mental health has commonly been evaluated by assessing care practices and interventions based on a set of clinical practice guidelines² (CPGs). However, these evaluation systems have been criticized for not taking the preferences of users into consideration³.

The generic instruments that exist for general evaluation of health services may not be relevant in a mental health context⁴ since mental health differs from other specializations regarding the interaction between patient and professional ascribed as a therapeutic tool, due to the element of compulsion in mental health care⁵.

There is no widely accepted definition of the concept of quality of care. Rather, the term refers to a multi-dimensional concept⁶ that is perceived by mental health care users as a positive concept, specifically in reference to good quality of care⁷. Unlike the concept of patient satisfaction, quality of care includes the perspectives of all the interested parties⁸. The experience of the mental health professional may be seen as something to bear in mind and may be used as an additional indicator of the quality of mental healthcare⁸. The essential components that make up this quality are the therapeutic setting, the therapeutic relationship and support, assessment, professional performance, assessment of practice, and environmental health⁹.

However, it has been demonstrated that the various professional disciplines involved in mental healthcare have differing points of view regarding exactly what characterizes quality in care¹⁰. The results of earlier studies¹¹ suggested that the perspective of nurses in relation to the quality of care was focused on interpersonal relationships, while other professionals were centered on organizational structures.

Western countries share an interest in evaluating and improving quality of care in the healthcare field¹². In spite of this, there is a lack of intercultural comparison of the perceptions of patients and staff regarding quality of care¹³. This is due primarily to the fact that cross-cultural research in psychiatric care lacks standardized instruments¹².

Instruments for measuring mental health care need to be valid and reliable. Some of the instruments used have been ad hoc in nature and have been criticized because their psychometric properties have rarely been documented¹⁴. Nevertheless, a review of the literature revealed a large number of patient satisfaction instruments for hospitalized mental health patients, although the psychometric properties of these instruments demonstrated variable results¹⁵.

The experience of the mental health professional should be seen as something to bear in mind and may be used as an additional indicator of the quality of mental healthcare¹⁶. Assessment of quality of care by professionals provides information on the therapeutic relationship formed with patients, the setting and its impact on care practice, and the relationships with teams and managers¹⁷. However, it has been demonstrated that the various professional disciplines involved in mental healthcare have differing points of view regarding what constitutes quality in care¹⁸. The study by Mason et al¹¹ suggested that the perspective of nurses in relation to the quality of care was focused on interpersonal relationships, while that of other professionals was centred on organizational structures.

To date, there has been no Spanish-language instrument for measuring the perspective of mental health professionals in terms of quality in psychiatry care. The Quality in Psychiatric Care-Inpatients Staff (QPC-IPS) instrument has the potential to fill this gap in research. This study is part of a larger research project to adapt the QPC-IPS to different international settings, test the psychometric properties and equivalence of dimensionality of the different language versions, and describe and compare the quality of inpatient psychiatric care across different countries.

Materials and methods

The objective of the present study was to describe the translation of the QPC-IPS into Spanish and the stages in its validation process and to test its psychometric properties.

A psychometric study was conducted, translating the QPC-IPS instrument into Spanish and assessing its psychometric properties, including content validity, convergent validity, construct validity, internal consistency, and test-retest reliability.

Adaptation of the Spanish instrument

Participants. The process of adaptation, translation and back-translation were carried out by a panel of experts made up of nurses, psychiatrists, a psychologist with expertise in psychometry, and care managers. After obtaining the definitive version of the instrument, a pilot test was carried out with 30 professionals from mental health hospital units.

Instrument. The perceptions of mental health professionals regarding the quality of care were obtained using the QPC-IPS instrument. The QPC-IPS consists of 30 items and measures six dimensions: Encounter (eight items), Participation (eight items), Discharge (four items), Support (four items), Secluded environment (three items), and Secure environment (three items). The QPC-IPS is based on the QPC-IP instrument by Schröder¹⁹. The definition was developed from a phenomenographic interview study⁷, and the instrument was tested for face validity in a pilot study and also empirically tested¹⁹.

Data collection. All of the items in QPC-IP were reformulated to adapt them to the context of staff working in psychiatric inpatient care. Each item begins with the sentence I experience that... and is assessed using a Likert-type scale with four options, going from 1 (totally disagree) to 4 (totally agree). The maximum score is 120 points and the minimum is 30 points. A high score in each dimension or for the scale as a whole indicates a perception of high quality on the part of the professional. In contrast, a low score would justify the need for intervention in the areas found to be lacking. For each item, there was also the possibility of answering not applicable. In addition, the questionnaire includes a number of background questions covering demographics and general clinical characteristics, and at the end of the questionnaire, there is an open-ended question inviting professionals to make comments regarding the quality of care being provided.

The original QPC-IPS instrument in Swedish was translated into Spanish using a translation/back-translation process. First, the Swedish version was translated into Spanish. The research group reviewed the translation and checked that the meaning of each item had been transmitted and translated correctly (cultural validation). Each item was rated on a scale of from 1 to 4 (minimum-maximum) with regard to its coherence, clarity, and relevance. Thereafter, the Spanish version was translated back into Swedish in order to check that the Spanish translation corresponded to the original Swedish text. The Swedish research group, the authors of the original QPC-IPS instrument, discussed the back-translation and compared it with the original Swedish versions to examine the convergence between the translation and the back-translation. After this discussion, the preliminary Spanish version of the QPC-IPS was created, paying attention to semantic equivalence (face validity). A pilot test was later conducted with the participation of 30 professionals, who underwent a cognitive interview (Fig. 1). They confirmed that the translated version of the instrument was easy to understand and complete.

QPC-IPS: Quality in Psychiatric Care - Inpatient Staff version; NTP 394 Job satisfaction: Overall Job Satisfaction Scale.

Figure 1.  Overview of the three-phase validation study. 

Results

For the adaptation of the Spanish instrument, a panel of experts produced the results based on coherence, clarity, and relevance greater than 3. No items required modification. After conducting a pilot testing and a cognitive interview with 30 professionals, it was determined that the QPC-IPS was adequate and could be self-administered. The results of this phase were positive, and there were no problems in the comprehension or administration of the questionnaire (Table 1).

The analysis of psychometric properties was carried out on responses obtained from 173 professionals. A total of ten returned questionnaires were excluded as they had 30% or more missing data. A total of 163 mental health professionals belonging to two hospitals (46.6 and 56.8%) in two districts of the province of Barcelona participated. The average age was 38.7 years (SD: 10.3), and 57.1% were women. In relation to nationality, 96.9% were Spanish. With regard to professional category, 39.9% were nurses and 47.2% nursing assistants. Psychology and psychiatry professionals accounted for 7.4% of the sample. More than half of the professionals worked in day-shift (60.1%). The mean number of years spent working in the current unit was 4.4 (SD: 5.6).

The mean item value ranged from 2.21 to 3.63, and the standard deviation ranged from 0.58 to 1.03. The item with the highest percentage ceiling rankings was item 15 (the staff respect the patients, 68.1%) and only two items presented floor ranking: item 2 (there is a high level of security at the ward, 25.2%) and item 28 (there is a private place where patients can receive visits from their next-of-kin, 25.8%) (Table 2).

The Cronbach's α internal consistency coefficient for the full scale was 0.92, with values above 0.70 being achieved in four of the six dimensions of the Spanish QPC-IPS (Table 3). In the Discharge and Secluded Environment dimensions Cronbach's α values of 0.62 and 0.66 were obtained, respectively. These levels are lower than desirable but close to 0.70. The α values were also calculated excluding each item or question from the scale, and no exclusion of an item was found to improve the internal consistency of the scale in a relevant way. All items had item-total correlations > 0.30.

The analysis of the ICC, shown in Table 2, demonstrated that test-retest reliability was 0.91 (95% CI: 0.87-0.94; n=92) and that the test was adequate for all dimensions except for the Discharge dimension, which scored 0.68 (95% CI: 0.52-0.79).

Convergent validity was analyzed using Spearman's correlation coefficient with the NTP394 General Satisfaction scale. The correlation obtained was rho=0.58 (p<0.0001).

The CFA revealed that the Encounter dimension and the Support dimension presented the largest factor loadings, while the Participation dimension and the Discharge dimension presented the lowest (Fig. 2). However, all of the loadings were statistically significant.

*: p < 0.05; LS: least squares.

Figure 2.  Factor loadings derived from the least square estimation (least squares). Confirmatory factor analysis (λij). 

All QPC-IPS dimensions showed high inter-correlations, except for the Support dimension, which scored below 0.50 with the Secluded environment and Secure environment dimensions (0.42 and 0.41 respectively).

The Chi-squared test was statistically significant but the chi-squared by degrees of freedom ratio was 2.08 (X²/df). As shown in Table 4, the other absolute fit, incremental fit, and the parsimony-based indices analyzed also showed a reasonably good fit.

Discussion

The objective of the present study was to describe the translation of the QPC-IPS into Spanish and the stages of the validation process, and to test the psychometric properties of the Spanish QPC-IPS.

During the adaptation process of the QPC-IPS, a translation and back translation was performed, obtaining an instrument adapted to Spanish. Other studies²⁹ of the QPC family have also obtained original instruments adapted with a similar process. The results of this phase were positive and there were no problems in the comprehension or administration of the questionnaire.

The results demonstrated that the psychometric properties in terms of internal consistency, temporal stability (test-retest), content validity, and construct validity (confirmatory factor analysis) were adequate. These results confirm that the structure of the Spanish version is similar to that of the original Swedish version of the QPC-IP.

The variability of most items was substantial, although most scores showed a ceiling or floor effect.

Cronbach's α for internal consistency was 0.92 for the full scale, with values close to or above 0.70 obtained for all dimensions. These values are considered adequate by Nunnally & Bernstein³⁰, and are higher than or very similar to those obtained in the original version¹⁹ and in other studies that have used other versions of the QPC instrument²⁹. The Discharge dimension obtained a Cronbach's α of 0.62 and the Secluded environment dimension an α of 0.66. These dimensions have shown less than adequate internal consistency in previous studies²⁹. Given the influence of several elements on Cronbach's α coefficient, the low α value in these dimensions is probably due to the small number of items (two and three items, respectively). It is also worth pointing out that the reliability results obtained for the Spanish QPC-IPS are similar to those for other quality of care measures reported in a recent systematic review¹⁵.

Temporal stability has not previously been analyzed on any QPC-instruments. The analysis of the Spanish version of QPC-IPS is the first. The ICC indicates good temporal stability. The Discharge dimension showed the lowest ICC values (ICC = 0.68). One of the reasons for these low ICC values may be that patient discharges from the ward were not scheduled in advance. This means that there are different types of discharges according to the care pressure, and therefore the professionals respond with different scores depending on the level of pressure. Another explanation may be that the mental health professionals give the patients information on discharge intermittently. This means that one week the mental health professional may have time to inform, and the next week not. This could account for the low values for the Discharge dimension.

In comparison to the instruments considered in a recent systematic review¹⁵, our results for Temporal stability were similar to those for the Spanish QPC-IP: the Combined Assessment of Psychiatric Environments instrument (CAPE)³¹, with a Pearson correlation coefficient of 0.82 for the staff version, the General Practitioner Experiences Questionnaire instrument (GPEQ)³², with an ICC of 0.72-0.87, and the Psychiatric Out-Patient Experiences Questionnaire (POPEQ)³³, with an ICC of 0.90.

The convergent validity of the Spanish QPC-IPS was examined by calculating Spearman's rho correlation coefficient with the NTP 394. A positive and moderate correlation was obtained with the NTP 394 as, according to Martínez et al³⁴, values between 0.31 and 0.70 present a moderate association. In this sense, as the results move in a positive direction, it may be inferred that the greater the job satisfaction, the more positive the perception of the quality of care is. This is the only study in which convergent validity has been analyzed with one satisfaction instrument.

With respect to construct validation, the CFA of the Spanish QPC-IPS showed the same factor structure as the original version of the QPC-IP, i.e., six dimensions of quality in mental health care⁷. In terms of the proposed criteria, from Worthington and Whittaker³⁵, the simple size was deemed sufficiently large to carry out CFA. It may be concluded that the dimensions of the Spanish QPC-IPS are identical to those of the original scale. In the Indonesian version of the QPC-IPS²⁹, the same six dimensions were observed. Thus, the concept of quality of mental health professionals from a Spanish perspective is similar to that among Indonesian mental health professionals. It is likely that mental health professionals are more observant about activities performed by themselves or other staff members compared to the patients, who may not fully observe what mental health professionals actually do.

The fact that the QPS-IPS has six dimensions is consistent with the assumption that the concept of quality is multidimensional⁷. In relation to the individual items, in all cases factor loading was greater than 0.40, which is generally considered to be the minimum level³⁶. The CFA revealed that all items presented adequate and statistically significant factor loading, resulting in adequate validity levels. The goodness-of-fit indices showed a reasonably good fit²⁷. These goodness-of-fit results were very similar to those found in previous studies of the Quality in Psychiatric Care-In-Patients¹⁴, the Quality in Psychiatric Care-Out-Patients³⁷, and the Quality in Psychiatric Care-Forensic In-Patients Staff instrument²⁹.

Finally, we should note that the mental health professionals in this study gave the highest quality rating to Support, which is in the line with previous studies on patients and mental health professionals²⁹ in an inpatient psychiatric care setting. The Encounter dimension was the second highest in terms of quality. Secluded environment was rated the lowest in terms of quality of care, as in the study by Lundqvist et al²⁹ performed in Indonesia. One possible explanation for this is that in Spain there are no individual rooms available, except for isolation rooms for emergency crises. However, it should be noted that the Secluded environment dimension in the Spanish QPC-IP¹⁴ fit the original Swedish model and the Indonesian model²⁹. This result indicates that the concept of secluded environment is more or less the same in Spain, Sweden, and Indonesia.

The limitations of this study are that QPC-IPS instrument was adapted to the inpatient context in Spain. Therefore, any generalized use of this adapted instrument must take into account that it should be applied to a similar population. Second, although the minimum sample required for the analysis was estimated²², the sample size was small. Finally, it should be noted that it was not possible to evaluate sensitivity to change or predictive validity, since the study design was transversal. This needs to be taken into account and addressed in the design of future studies.

The Spanish QPC-IPS is a simple and easily administered tool for measuring various aspects of quality in psychiatric inpatient care from the perspective of mental health professionals. Its six-factor structure and psychometric properties are consistent with those of the original instrument, lending support to its use to measure quality of care in Spanish-speaking populations. The results of such measurements could be used to improve the quality of the service provided. Future studies will need to look at the psychometric properties of this instrument in relation to other variables and other samples of mental health professionals, both in the community and in other settings.