Original Research

Validation of a survey tool for use in cross-cultural studies

Filipa A. COSTA, Catherine DUGGAN, Ian BATES.

ABSTRACT There is a need for tools to measure the information patients need in order for healthcare professionals in general, and particularly pharmacists, to communicate effectively and play an active part in the way patients manage their medicines. Previous research has developed and validated constructs to measure patients’ desires for information and their perceptions of how useful their medicines are. It is important to develop these tools for use in different settings and countries so that best practice is shared and is based on the best available evidence. Objectives: this project sought to validate of a survey tool measuring the “Extent of Information Desired” (EID), the “Perceived Utility of Medicines” (PUM), and the “Anxiety about Illness” (AI) that had been previously translated for use with Portuguese patients. Methods: The scales were validated in a patient sample of 596: construct validity was explored in Factor analysis (PCA) and internal consistency analysed using Cronbach’s alpha. Criterion validity was explored correlating scores to the AI scale and patients’ perceived health status. Discriminatory power was assessed using ANOVA. Temporal stability was explored in a sub-sample of patients who responded at two time points, using a T-test to compare their mean scores. Results: Construct validity results indicated the need to remove 1 item from the Perceived Harm of Medicines (PHM) and Perceived Benefit of Medicines (PBM) for use in a Portuguese sample and the abandon of the tolerance scale. The internal consistency was high for the EID, PBM and AI scales (alpha>0.600) and acceptable for the PHM scale (alpha=0.536). All scales, except the EID, were consistent over time (p>0.05; p<0.01). All the scales tested showed good discriminatory power. The comparison of the AI scale with the SF-36 indicated good criterion validity (p<0.05). Conclusion: The translated tool was valid and reliable in Portuguese patients- excluding the Tolerance scale. Some of the scales may benefit from further refinement, such as the PHM subscale. Key words: Cross-Cultural Comparison. Validation Studies as Topic. Portugal.

RESUMEN Se necesitan herramientas para medir la información que necesitan los pacientes para que los profesionales de la salud en general, y los farmacéuticos en particular, comuniquen efectivamente y jueguen un papel activo en como los pacientes manejan su medicación. Anteriores estudios han desarrollado instrumentos para medir los deseos de los pacientes de información y sus percepciones sobre la utilidad de los medicamentos. Es importante desarrollar estas herramientas en diferentes escenarios y países para compartir la mejor práctica y que se base en la evidencia disponible. Objetivos: Este proyecto trató validar una herramienta de encuesta que mide la “Cantidad de Información Deseada” (CID), la “Utilidad Percibida de la Medicación” (UPM), y la “Ansiedad sobre la Enfermedad” (AE) que fueron previamente traducidos para el uso con pacientes portugueses. Métodos: Las escalas se validaron en una muestra de 596 pacientes: la validez de constructo fue explorada en un análisis factorial y la consistencia interna se analizó usando un alfa de Cronbach. La validez de criterio se exploró correlacionando las puntuaciones a la escala AE y al estado de salud percibido de los pacientes. El poder discriminante se evaluó utilizando un ANOVA. La estabilidad temporal se exploró en una sub-muestra de pacientes que respondieron en dos momentos, usando un test-T para comparar sus puntuaciones medias. Resultados: La validez de constructo indicó la necesidad de retirar un ítem de Daño Percibido de los Medicamentos (DPM) y de Beneficio Percibido de los Medicamentos (BPM) para el uso en una muestra portuguesa y el abandono de la escala de tolerancia. La consistencia interna fue alta para las escalas CDI, UPM y AE (alfa>0,06) y aceptable para DPM (alfa=0,536). Todas las escalas, excepto CID fueron estables en el tiempo (p>0.05; p<0.01). Todas las escalas probadas mostraron alta capacidad discriminante. La comparación de la escala AE con el SF-36 indicó una buena validez de criterio (p<0,05). Conclusión: La herramienta traducida fue valida y fiable en pacientes portugueses, excluyendo la escala de tolerancia. Algunas de las escalas pueden beneficiarse de un refinamiento posterior, como la sub-escala DPM. Palabras clave: Comparación trans-cultural. Estudios de validación como tema. Portugal.

---

Filipa Alves da COSTA. MSc, PhD. Auxiliary Professor, Instituto Superior de Ciências da Saúde, ISCSEM. Caparica (Portugal).
Catherine DUGGAN, BPharm(Hons), PhD, MRPharmS. Senior Clinical Lecturer - School of Pharmacy, University of London, and Associate Director for Clinical Pharmacy, Development and Evaluation for East and South East England Specialist Services (NHS). London (United Kingdom).
Ian BATES. BPharm, MSc, FRSH, MRPharmS. Professor and Head of Educational Development. School of Pharmacy, University of London. London (United Kingdom).

Received (first version): 17-Mar-2008
Accepted: 20-Jun-2008

 

INTRODUCTION

The development of standardised scales to assess the information each patient wants can be of major importance to effectively target health care interventions. A survey tool has been developed to explore patients’ desires for information and their perceptions about prescribed therapy and illness. The initial tool, comprising 5 constructs, has been used in several studies and has been shown to describe and predict patients’ information needs and perceptions of their medicines. Further validation of three of these constructs has led to factors describing the “Intrinsic Desire for Information” (IDI), the “Perceived Utility of Medicines” (PUM), and the “Anxiety about Illness” (AI).² Additional work on the initial 12-item IDI scale resulted in a 6-item scale: “Extent of Information Desired” (EID).^1,3

An important attribute when developing a practical scale is its length; this scale uses the minimum number of items comprising three important dimensions: the way the individual perceives his illness; the way he perceives his medicines; and takes into account the importance of patient-health care professional communication. Considering the potential utility of such tools for health-care professionals to increase their awareness of patients’ desires, perceptions and feelings, the applicability of such a scale in other countries and settings was evaluated.

There is a problem when directly applying scales developed in English speaking countries for use in non-English speaking countries (eg: English scales for use in Portugal) and will increase as more countries enter the European Union. Networks to foster collaborative work are established across Europe to share and compare data.⁴ It is therefore important to adapt scales in a robust way for use across cultures that can assist health care professionals to target interventions to improve patient care.

The aim of this study was to validate a survey tool to measure patients’ information desires and perceptions about medicines and illness in a Portuguese patient sample.

 

METHODS

Ethics

Ethical approval had previously been obtained for the UK data. In Portugal, no official ethics approval systems were in place applicable to this study design and content, so a written authorisation was obtained from the head of each service or hospital, accordingly.

Study design

An initial exploratory study using a cross-sectional design evaluated two subsequent versions of the translated tool in regards to validity and internal consistency. The tool’s consistency was tested over time through a test-retest longitudinal study.

Sample

Previous research showed that 5 factors emerged from the pool of 50 items. The sample size was calculated to be 250 recruits for validation purposes and another 200 for purposes of test-retest reliability analysis. The latter sample would detect differences between mean scores to the scales for a 95% confidence interval (5% error and an 80% power). This paper describes part of a larger study but all data collected by the time of submission is presented.

A pilot study was undertaken in GEM patients to ensure that data collection materials and methods were appropriate for the main study. The main study involved testing the final version in an independent medical patient sample. The inclusion criteria restricted participants to adult patients prescribed with medication, understanding written and/or spoken Portuguese and agreeing to participate in the study once informed. Patients were invited to participate and were then provided with additional information about the research by means of a written invitation. Prior to the interview, patients read and signed an informed consent form.

Participants recruited in the hospital setting were interviewed at their bedside using a standardised questionnaire, comprising three main sections: demographic data, diagnoses and prescribed medicines; the survey tool’s items, which were statements to be answered using a 5-point Likert scale, ranging from strongly agree to strongly disagree; and a set of open questions to address the concepts explored through the scales. In the community pharmacy setting, questionnaires were self-administered while similar in content. Patients who were unwilling to participate were given a refusal form, seeking to explore different characteristics of non-respondents.

Data analysis

Data were analysed using descriptive statistics to characterise the sample and responses to the scales. The Kolmogorov-Smirnov Test was used to explore the distribution of scores to scales before deciding on the use of parametric or non-parametric tests. Construct validity was explored in factor analysis, using Principal Components Analysis (PCA) with Oblimin rotation. The emerging subscales were subsequently compared with retrospective UK data. Criterion validity was explored by correlating scores to the anxiety scale and patients’ responses to the SF-36 general question included in the questionnaire using Spearman’s rho. The tool’s discriminatory power was assessed using ANOVA or independent samples t-test, as appropriate. Internal consistency was analysed using Cronbach’s coefficient alpha and inter-item correlations (r) and again compared with the UK retrospective data. Test-retest reliability was explored in a sub-sample of patients who responded to the questionnaire at two time points one month apart. Data were treated as paired samples, using a t-test to compare their mean scores.

 

RESULTS

A sample of 62 patients was recruited for the pilot phase. This highlighted difficulties in recruitment associated with some of the study sites where access was restricted to acute care wards. Therefore, inclusion criteria were reviewed to widen the recruitment possibilities.

Table 1 summarises the demographic, medical and therapeutic characteristics of the 596 cases included in the sample. Gender was evenly distributed; the mean age was as expected given the focus on chronically ill patients. There was a high proportion of people without education and an extremely high proportion of individuals with primary school education. Cardiovascular and endocrine patients (57%) were the most common diagnoses. The number of non-respondents was very low (n=25; 4%), hence no statistical tests were performed to evaluate differences between the two samples.

Validity of the scales

Criterion validity was judged to be good. A negative and statistically significant correlation was found when comparing the AI scale with the external measure (rho=-0.318; p<0.001), indicating that patients who perceived their health status as worse were more anxious.

Construct validity: Principal components analysis resulted in 6 factors, rather than the predicted 5 (table 2). Factor 6, when compared to the original tool, was judged to be the combination of those items that did not “work” in a Portuguese sample, despite modifications during translation. This factor comprised the items A4 (I just want to blame someone for the way I feel) and T1 (I find my medicines easy to take, I am used to them).

Most items fell into the expected domains for the AI and EID scales. The Tolerance scale (Ti), a subscale of the “Anxiety about Illness” scale originally had weak internal consistency (alpha=0.470) and contained only 3 items; one of which did not load into this factor and was then dropped, leaving this scale with only two items, insufficient to form a scale. As such, the Ti scale was dropped and considered unsuitable for the Portuguese population.

The items of the PUM scale loaded well into the expected sub-factors. The exception was the previously described item T1 and was dropped. Both the PHM and PBM subscales comprised only 3 items, rather than the UK sample 4. The version to be used in Portugal had therefore the tolerance scale removed as well as one item of the PUM scale. The final scale comprised a total of 16 items.

Reliability of the scales

Both the EID and PHM scales showed acceptable internal consistency (alpha=0.607 and 0.536, respectively), which could be further improved by additional modifications. The AI and the PBM scales revealed good internal consistency estimates (alpha=0.695 and 0.756, respectively). In all scales, the removal of any item would result in a decrease of its overall internal consistency. Table 3 displays the Cronbach’s alpha values and the inter-item correlations range for each scale and item in the Portuguese sample and their comparison with retrospective UK data.

All scales, except the EID were consistent over time, since their mean scores did not differ between baseline and after one month. The correlation between the PBM, PHM, and AI scores (at baseline and after one month) was high and statistically significant, indicating the concepts are being measured consistently. Furthermore, their mean scores were not significantly different, confirming these scales are consistent over time. Scores to the EID scale were strongly correlated between the two time points, but mean scores were significantly higher at baseline (25.9) than after one month (24.2) which implies patients seem to want less information as time goes on- maybe they are more “used” to their medicines by this stage

Discriminatory power

The scores to the four valid scales were explored in samples of inpatients, outpatients and community pharmacy patients. Significant differences between mean scores to scales were detected for the EID scale between patients in the three settings (F=9.891; p<0.001) and the AI scale (F=4.916; p=0.008). This implies patients recruited from different study sites had different desires for information and levels of anxiety about their illness.

To ensure that the setting had no further influence on the scales’ discriminatory power, analysis was restricted to community pharmacy patients. Females scored significantly higher than males to the PBM scale (t=-2.174; p=0.031). Age was negatively correlated with scores to the EID scale (r=-0.24; p<0.001) and positively correlated with scores to the AI scale (r=0.16; p=0.001), both correlations weak but statistically significant. Categorizing age into two broad groups additionally showed that younger patients expressed a higher desire for information (t=4.19; p<0.001) and lower anxiety about their illness (t=-2.00; p=0.047).

The mean scores to the EID, PHM and the AI scales varied significantly between the lowest educational level and all others (EID: F=3.92; p<0.001); (PHM: F=2.10; p=0.011); (AI: F=5.18; p<0.001). The desire for information increased, the perception of harm and anxiety decreased as years of formal education increased. Patients prescribed 5 or more drugs scored significantly higher on the Anxiety scale (t=-3.22; p=0.001). It may be assumed that when patients are prescribed more drugs, they interpret this as a signal of deterioration in their condition, hence increasing their anxiety.

In summary, the translated survey tool was found to be both valid and reliable after being tested in a patient sample of 596 patients with a wide representation of demographic and medical characteristics. The sample reflected the general characteristics of the Portuguese population well.

The five scales described in the original UK research behaved differently in the Portuguese population following translation. The tolerance scale was dropped during the course of validation work as it was deemed to be unsuitable for use in this patient sample. Additionally, the two subscales in the PUM scale were transformed by dropping one common item as it did not load into any of the expected factors and internal consistency of both scales increased when it was deleted. Both the EID and the AI scales were left unaltered and found to have good construct validity.

• The EID scale showed to have an acceptable internal consistency (alpha=0.607) but not being consistent over time (p<0.05; p<0.01). The EID scale was able to discriminate between settings, age group and educational levels (p<0.05).

• The PBM scale showed to have a good internal consistency (alpha=0.695) and being consistent over time (p>0.05; p<0.01). Construct validity results led to dropping one of the items, whilst not compromising its reliability. This scale showed to discriminate between different genders (p<0.05). The distribution of scores to the PBM scale showed small spread, indicating a possibility for little added-value in its use.

• The PHM scale was the one with the lowest internal consistency (alpha=0.536), considered fair, but being consistent over time (p>0.05; p<0.01). Construct validity results led to dropping one of the items, whilst not compromising its reliability. This scale showed to discriminate between educational levels (p<0.05).

• The AI scale was the one with the highest internal consistency (alpha=0.756), considered good, and also showing consistency over time (p>0.05; p<0.01). This scale showed good results in regards to criterion validity, when compared with the SF-36 (p<0.05). Additionally it showed to discriminate between settings, age group, educational levels and number of prescribed drugs (p<0.05).

 

DISCUSSION

The characteristics of the sample were compared to national data, where illiterate people represent 8% of the population (compared to 7% in the study sample) and people with university education account for 11% (compared to 12% in this sample). Study recruits with primary school education accounted for 47% of the sample, compared to 35% in the national population. Nonetheless, it may be concluded that the sample educational level represents the Portuguese population fairly well.⁵

Internal consistency of the EID was not as high as expected from previous UK studies.^1-3 A Cronbach’s alpha value of 0.7 has been suggested as good⁶ and the fact that the value reported for the EID in the Portuguese sample roughly passes 0.6 is an issue of concern. However, this scale has been further tested in another study including 112 hypertensive patients recruited from Lisbon pharmacies, where the value obtained was 0.683.⁷ This increase may be related to distinct characteristics of the samples, the latter study was conducted in a particularly developed area, where educational levels are generally higher. This may imply that this scale’s performance may improve as the Portuguese population’s literacy increases. This trend is supported by successive legislation reviewing the minimum compulsory education.⁸

Consistency over time for the EID scale was not detected which may be due to the changes made just by asking the patients; and possible interventions made by the pharmacists while administering the questionnaire, simply by being aware of patients’ desires. Alternatively, information needs may be easily changed dependent on situations, time, severity of symptoms, whereas anxiety and perceptions of medicines may be less so.

Face and content validity have also been examined, albeit reported elsewhere⁹, where problems were revealed for the EID, PHM and Ti scales. The latter was dropped; the others may need additional work when being tested in practice. Additional problems were encountered with the Anxiety scale as it led to anguish in some patients, which could potentially compromise its use in pharmacy practice. Nonetheless, it may be worth exploring its applicability in medical settings, where scales such as the Beck’s Depression Inventory (BDI) have been employed.¹⁰

Criterion validity is often not reported due to lack of comparable measures in this area (which further validates the need to develop them). In this study, criterion validity was explored in three different ways, using qualitative data, interfacing qualitative and quantitative data and external measures to correlate with some of the scores to scales, although only the latter was expanded in this manuscript. Other external measures could have been used for comparison with the PUM scale, such as the BMQ.¹¹ This would have undisputed advantages, but also an additional burden for respondents which could potentially have an impact on the response rate, so was disregarded in this study. Additionally, this questionnaire is not validated to Portuguese. There is no tool comparable to the EID that could have been used, leaving patients’ own words as the only possible option.

Some of the limitations of this study include not having used an external measure for each scale to explore criterion validity, limiting quantitative analysis of this type of validity to the Anxiety scale. The limited length of hospital stay resulted in the impossibility of having repeated the questionnaire also in hospital patients to explore consistency over time, which would have the advantage of ensuring the same interviewer, minimising probabilities of interviewer-related bias.

Selection bias in community pharmacy occurred since participating pharmacists had to combine participating in a research study with minimal impact on normal service delivery. The inability to guarantee response of refusal forms limited the usefulness of the analysis undertaken to explore comparability of respondent and non-respondents, but this was a limitation impossible to overcome as people cannot be forced to answer.

 

CONCLUSIONS

The main conclusions are that the translated tool was valid and reliable in Portuguese patients- excluding the Tolerance scale. Some of the scales may benefit from further refinement, such as the PHM subscale. Successive modifications are needed to increase the tool’s validity and reliability and confirm that scales are not directly transferable from one country to another. This study demonstrates the importance of robust translation and validation processes to measure the concepts of interest in a credible manner.

 

ACKNOWLEDGEMENTS

The authors acknowledge all the patients interviewed, the hospital staff, and all the community pharmacists who actively participated in patient recruitment.

 

CONFLICT OF INTEREST

The authors declare they have no conflict of interests. This project was entirely funded by CEFAR (Centre for Pharmacoepidemiological Research), ANF (National Association of Pharmacies), Portugal.

 

References

1. Aström K, Carlsson J, Bates I, Webb DG, Duggan C, Sanghani P, McRobbie D. Desire for information about drugs. A multi-method study in general medical inpatients. Pharm World Sci. 2000;22(4):159-164.         [ Links ]

2. Duggan C, Bates I. Development and evaluation of a survey tool to explore patients' perceptions of their prescribed drugs and need for drug information. Int J Pharm Pract 2000; 8(1): 42-52.         [ Links ]

3. Duggan C, Bates I, Sturman S, Andersson E, Astrom K, Carlsson J. Validation of a 'desire for information' scale. Int J Pharm Pract. 2002;10(1):31-37.         [ Links ]

4. PCNE. Pharmaceutical Care Network Europe. Available at www.pcne.org assessed 22/05/2006        [ Links ]

5. Instituto Nacional de Estatística. 2003, "Recenseamento Geral da População e Habitação (2001) - Resultados Definitivos." Lisboa. Data File ("General Licensing of the Population and Housing - Final Results") (original report in Portuguese).         [ Links ]

6. DeVellis RF. Factor analysis. Scale development, theory and applications. London: SAGE Publications, 2003: 102-137.         [ Links ]

7. Costa F, Couto S, Ferreira AP, et al. 2005a, "Hypertensive patients' perceptions about prescribed medicines. Proceedings of the FIP Congress (Abstract 108; CPS-P-083). Presented at the FIP Congress 2005        [ Links ]

8. Ministério da Educação. Portaria. Diário da República Portuguesa. 27-5-2005. n º 41/2005 ("Education Ministry. Law ") (original in Portuguese).         [ Links ]

9. Costa FA, Duggan C, Bates I. A systematic approach to cross-cultural adaptation of survey tools. Pharm Pract. 2007;5(3):115-124.         [ Links ]

10. Coelho R, Ramos E, Prata J, Barros H. Psychosocial indexes and cardiovascular risk factors in a community sample. Psychother Psychosom. 2000;69(5):261-74.         [ Links ]

11. Horne R, Weinman J, Hankins M. The beliefs about medicines questionnaire: the development and evaluation of a new method for assessing the cognitive representation of medication. Psychol Health 1999;14:1-24.         [ Links ]