Original Research

Development and validity of a method for the evaluation of printed education material

Mauro Silveira de CASTRO, Diogo PILGER, Flávio Danni FUCHS, Maria Beatriz Cardoso FERREIRA.

ABSTRACT Objectives: To develop and study the validity of an instrument for evaluation of Printed Education Materials (PEM); to evaluate the use of acceptability indices; to identify possible influences of professional aspects. Methods: An instrument for PEM evaluation was developed which included tree steps: domain identification, item generation and instrument design. A reading to easy PEM was developed for education of patient with systemic hypertension and its treatment with hydrochlorothiazide. Construct validity was measured based on previously established errors purposively introduced into the PEM, which served as extreme groups. An acceptability index was applied taking into account the rate of professionals who should approve each item. Participants were 10 physicians (9 men) and 5 nurses (all women). Results: Many professionals identified intentional errors of crude character. Few participants identified errors that needed more careful evaluation, and no one detected the intentional error that required literature analysis. Physicians considered as acceptable 95.8% of the items of the PEM, and nurses 29.2%. The differences between the scoring were statistically significant in 27% of the items. In the overall evaluation, 66.6% were considered as acceptable. The analysis of each item revealed a behavioral pattern for each professional group. Conclusions: The use of instruments for evaluation of printed education materials is required and may improve the quality of the PEM available for the patients. Not always are the acceptability indices totally correct or represent high quality of information. The professional experience, the practice pattern, and perhaps the gendre of the reviewers may influence their evaluation. An analysis of the PEM by professionals in communication, in drug information, and patients should be carried out to improve the quality of the proposed material. Key words: Patient Education. Comprehension. Validation studies. Brazil.

RESUMEN Objetivos: Desarrollar y estudiar la validez de un instrumento para la evaluación del material educativo impreso (MEI); evaluar el uso de los índices de aceptabilidad; identificar las posibles influencias de los aspectos profesionales. Métodos: Se desarrolló un instrumento para evaluación del MEI en tres pasos: identificación de dominios, generación de ítems y diseño del instrumento. Se desarrolló un MEI fácil de leer para la educación de pacientes con hipertensión sistémica y su tratamiento con hidroclorotiazida. Se midió la validez del instrumento mediante los errores previamente introducidos a propósito en el MEI, lo que sirvió de grupo extremo. Se aplicó un índice de aceptabilidad teniendo en cuenta la tasa de profesionales que tenía que aprobar cada ítem. Participaron 10 médicos (9 hombres) y 5 enfermeras (todas mujeres). Resultados: Muchos profesionales identificaron los errores intencionales de carácter crudo. Pocos participantes identificaron los errores que necesitaban una evaluación más cuidadosa, y ninguno identificó el error intencional que requería análisis de la literatura. Los médicos consideraron aceptable el 95,8% de los ítems del MEI y las enfermeras el 29,2%. Las diferencias de puntuación fueron estadísticamente significativas en el 27% de los ítems. En la evaluación total, el 66,6% fue considerado aceptable. El análisis de cada ítem reveló un modelo de comportamiento para cada grupo de profesionales. Conclusiones: Es necesario el uso de instrumentos para la evaluación de materiales educativos impresos y puede mejorar la calidad de los MEI disponibles para los pacientes. Los índices de aceptabilidad no son siempre totalmente correctos ni representan la calida de la información. La experiencia profesional, el modelo de ejercicio, y quizás el género de los revisores puede influir en la evaluación. Debería realizarse un análisis de los MEI por profesionales de comunicación, de la información sobre medicamentos y por pacientes para mejorar la calidad del material propuesto. Palabras clave: Educación de pacientes. Comprensión. Estudios de validación. Brasil.

---

Mauro Silveira de CASTRO. PhD. Departament of Drug Production and Control, School of Pharmacy. Universidade Federal do Rio Grande do Sul (Brazil).
Diogo PILGER. MSc. Post-graduate Program of Epidemiology. Universidade Federal do Rio Grande do Sul (Brazil).
Flávio Danni FUCHS. PhD. Division of Cardiology, Hospital de Clínicas de Porto Alegre (Brazil).
Maria Beatriz Cardoso FERREIRA. PhD. Departament of Pharmacology - Universidade Federal do Rio Grande do Sul (Brazil).

 

INTRODUCTION

Printed education material (PEM) is one of the most frequently used resources for the education of patients with chronic diseases. Different authors have established the most important steps in developing PEM^1-10 for literate patients. Some articles in the current literature have reported the poor quality of information of such materials^11-16, as well as the lack of scientific accuracy to promote the education of patients.^11,12,16-23

Printed education materials have been used routinely in Brazil for patients from the public and private institutions, but with no defined validation process. Two reports on evaluation of such materials for Brazilian patients were found.^24,25 The first focuses on criteria for evaluation of textbooks for elementary education²⁴ and the second evaluates the acceptability of one piece of education material produced in the form of a game.²⁵ The lack of assessment of PEM produced in the Portuguese language and the new requirements for the development of PEM^17,23 demonstrate that it is necessary to develop further research in the educational area. In this report, the development and testing of an instrument for evaluation of PEM (EVALPEM) for health professionals and the performance of the acceptability index of the evaluated items, and the influence of some professional aspects in this process are presented.

 

METHODS

Development of the evaluation tool

The steps proposed by Lynn² were used for identification of domains, generation of items and tool design.

Phase 1: Identification of domains

Domains were those identified in the literature review or regarded as important for determining the quality of a PEM, according to expert evaluation.^3,17,23,26

Phase 2: Items Generation

The items were established for each domain (described in Table 1), specifying the basic principles pertinent to each item.^{3,16,17,23,26}

Phase 3: Design of the Instrument

Based on the previously established domains and their respective items, a tool for the evaluation of PEM was developed. Each item was presented in the form of an affirmation, followed by an assessment scale with 3 categories: totally disagree, corresponding to 0 points; partially agree, corresponding to 1 point; totally agree, corresponding to 2 points. This scale was adapted from the scale proposed in The Bernier Instructional Design Scale (BIDS).³ After the scoring scale, there was a blank space for comments. In the instructions, the evaluators were asked to report any problems faced in this field while setting up the PEM or to state "not applicable" when appropriate. At the end of the EVALPEM, four questions were asked, aiming an integral evaluation of the PEM: 1) What did you like in the PEM? 2) What did you not like in the PEM? 3) What should be added? 4) What should be reviewed?

Construct validity

It was not possible to validate EVALPEM as proposed by Bernier³ and Coulter et al.¹⁷ since it was not possible to find a sufficient number of experts in patient education in Brazil. Therefore, we decided to evaluate the validity of the designed instrument using an alternative method. The introduction of errors previously established and introduced in a PEM constructed for this purpose was the artifice used, thus providing correct and incorrect elements that acted as extreme groups.²⁷ Through the analysis of the "extreme groups," we intended to verify whether the evaluators would be capable of detecting errors using the proposed instrument, as well as identifying possible professional influences when applying a tool like EVALPEM.

The PEM consisted of two blocks, one related to systemic arterial hypertension and the other to the use of hydrochlorothiazide. These themes were chosen due to the high prevalence of hypertension in our city^28,29 and the use of hydrochlorothiazide as one of the first choice medications for the management of this condition.^29-31 A version which required less than 6 years of schooling from patients was utilized.³² The basic structure of the PEM was previously presented.^17,33

Procedure for application of the EVALPEM

Health professionals that provided outpatient services in the Hospital de Clínicas de Porto Alegre and who were involved in the care of patients with hypertension were invited to evaluate the PEM. All of them agreed to participate of the study, not taking into account their real objectives or the existence of extreme groups.

For analytical purposes, they were divided into 2 groups. The first (Group 1) consisted of 10 physicians (9 men) from the Division of Cardiology. The second (Group 2) consisted of 5 nurses (all women) of the Nursing Service in Public Health. They received a copy of the PEM and the evaluation tool, as well as written instructions for completing it. After one week, the materials were returned with the questionnaires completed. The data from the two groups of professionals were analyzed separately and jointly in order to identify differences associated with their background.

Evaluation of the acceptability index of the proposed material

To evaluate the contents of the PEM, the parameters described by Lynn² were adopted. These parameters state that the acceptability of a given criterion depends on the approval of a certain proportion of experts. This proportion varies according to the numbers of evaluators. In Group 1, an item was considered acceptable when at least 80% of the possible points were obtained; in Group 2, 100% of the points were required to approve the item. The score for both groups combined was set in 80%.

 

RESULTS

In the process of construct validity, the physician and the nursing groups identified, respectively, 6 and 7 from 9 intentional errors. For two items only one professional identified the errors: 1) the absence of report concerning the adequate arm position in the measurement of blood pressure, and 2) the inadequate spacing between lines. The intentionally false contraindication of hydrochlorothiazide in patients with rheumatoid arthritis was not identified by any professional.

Based on the previously established parameters, it was observed that for the physician group 95.8% of the items in the PEM were considered acceptable. Six items were considered unacceptable. Four of them corresponded to the errors deliberately introduced in the PEM. The 2 other corresponded to the items that physicians did not either totally or partially agree with. The description of the mechanism of the action of the drug was considered difficult for patients to understand, and the description of adverse effects of hydrochlorothiazide was considered unnecessary.

For the nurse group, just 29.2% of the items were considered as acceptable. Forty items were considered as unacceptable, 7 of them referring to the deliberate errors that were adequately identified. In relation to the others, a behavior similar to the physicians was observed in regard to the amount of information to be provided to the patient. Moreover, nurses were more critical, indicating inadequacies in the PEM that were not reported by the physicians, such as double meaning expressions and usage of terminology considered difficult to understand by the patients.

The differences between the scoring of the 2 groups were statistically significant in 27% of the items (P < 0.05, Mann-Whitney U test). In the overall evaluation, 66.6% of the items were considered as acceptable.

The analysis of each item revealed a behavioral pattern for each professional group. Among the physicians, 7 suggested modifications in the PEM, such as to use this opportunity to better explain the risks for arteriosclerosis, to teach how to recognize angina and stroke. One physician suggested an investigation of patients in order to determine the best way to orient them about the administration of hydrochlorothiazide. Five physicians were against at least one of the following items: to inform patients about the stages of hypertension; to measure BP outside the medical office; to inform about potential adverse drug reactions; to require the control of blood glucose before starting the use of hydrochlorothiazide.

Four nurses gave suggestions: to extend the concept of hypertension; to highlight titles in a different color from the printed text; to identify the producers and reviewers in the PEM; and to better explain the relationship between loss of potassium and cramps. They also suggested that some points should be directly investigated with the patients. Four nurses were against at least one of 8 points, for example, the explanation of the classification of hypertension and the orientation on how to prevent intercurrences, since they might scare patients and reduce concordance of the treatment.

Some comments from the nurses were in disagreement with the recommendations from the literature, such as (1) disagreement regarding to diet orientation; (2) the possibility of controlled alcohol and salt consumption, opting for prohibiting; (3) the explanation on how to conduct in face of lack of one dose of the drug, since it could lead to confusion, and (4) the risk of photo-sensitivity associated with the use of hydrochlorothiazide.

Contrasting ideas on the same topic were also observed. A physician considered the material too simple while 3 nurses stated that the material was too long. Three physicians and a nurse reported their concern about the participation of pharmacists in the education of patients. Three physicians and four nurses did not agree to warn patients about potential adverse reactions to hydrochlorothiazide and deemed unnecessary for patients to know the stages of hypertension.

 

DISCUSSION

The quality of information provided to patients unequivocally contributes to the prevention of illnesses and recovery of health.^34-36 The quality of such materials should be evaluated. Previously to their use by patients, PEM needs to be evaluated by different groups of professionals involved with their care. Therefore, the development of instruments for this evaluation is required, such as the presented and tested in this report.

The questionnaire applied in this study was detailed and some items were considered irrelevant by the professionals. Both groups identified correctly the most evident deliberate errors. However, just a few professionals were able to identify the errors that would require a deeper analysis of the sources of information or about legibility and printing characteristics. These findings suggest that, in the domain of scientific accuracy, experts in the analysis of scientific literature should also review PEM information, as recommended by Coulter.¹⁷ Additionally, professionals of the communication area should be consulted about the illustrations, legibility and printing characteristics of the material.

The EVALPEM, which was tested in the evaluation of the printed educational material on hypertension and the use of hydrochlorothiazide, allowed identifying the opinions of physicians and nurses about the PEM. The physicians accepted most part of what was reported in the PEM, while the nurses did not accept many items.

The items that have discordant opinions from these two groups should be studied. Part of the different evaluation by physicians and nurses may be attributed to the different knowledge about the theoretical background topic by the professionals. Physicians were in charge of an outpatient clinic of hypertension, while the nurses that were invited to participate came from the Nursing Service in Public Health Hospital. We observed that much of the disagreement regarding the proposed PEM derive from their models of professional practice (The Medical Model versus The Helping Process)³⁴ and from the clinical decision making.¹⁷ Professionals tend to establish a relation of dependence rather than partnership with their patients. Oliveira reported similar results³⁷ in a study about the communication process between patients and physicians when a serious prognosis should be given to patients. Differences related to the gender could not be also discarded, since all nurses were women and most physicians were male.

We suggest that isolated remarks should not be discarded. Sometimes just one individual perceives the problems regarding to one topic, while all other evaluators approve it. These data reveal the limitation of using isolated acceptability indices in the PEM evaluation.

On the other hand, the fact of having consensus among the professionals on a particular item does not mean necessarily adequacy. Both groups, for example, disagreed on presenting potential drug adverse effects, a recommendation not supported by the best evidence.(38-43) The PEM improved on the basis of the evaluation by physicians and nurses should be sequentially evaluated by professionals of communication and by the patients themselves.

The reaction to the participation of pharmacists in patient education expressed by some physicians and nurses probably reflects the model of professional pharmaceutical practice adopted in Brazil, which is primarily oriented to drug distribution. The adoption of a new paradigm – pharmaceutical care – one of the World Health Organization recommendations^44,45, might contribute to solving health issues related to the use of medicines.

 

CONCLUSION

The present study demonstrated that printed education materials require the use of an evaluation process to improve their quality. Variables such as the evaluation instrument, the profile of reviewers, and the acceptability index should be used appropriately.

 

ACKNOWLEDGEMENT

Research sponsored by: Fundação de Amparo à Pesquisa do Estado do Rio Grande do Sul (FAPERGS), Fundo de Incentivo à Pesquisa do Hospital de Clínicas de Porto Alegre (FIPE-HCPA), Conselho Nacional de Pesquisa (CNPq).

 

References

1. McCabe BJ, Tysinger JW, Kreger M, Currwin AC. A strategy for designing effective patient education materials. J Am Diet Assoc 1989; 89:1290-1292.         [ Links ]

2. Lynn MR. Determination and qualification of content validity. Nurs Res 1986; 35:382-385.         [ Links ]

3. Bernier MJ. Establishing the psychometric properties of a scale for evaluating quality in printed education materials. Patient Educ Couns 1996; 29:283-299.         [ Links ]

4. Bernier MJ. Developing and evaluating printed education materials: a prescriptive model for quality. Orthop Nurs 1993; 12: 39-46.         [ Links ]

5. Brockett RG. Developing written learning materials: a proactive approach. Lifelong Learn Adult Years 1984; 7:16-18.         [ Links ]

6. Blacklay A, Eiser C, Ellis A. Development and evaluation of an information booklet for adult survivors of childhood cancer. Arch Dis Child 1998; 78:340-347.         [ Links ]

7. Moore MB, Sorensen M, Adebajo CF. Materiales impresos ilustrados para la educación en salud y planificación familiar. Foro Mund Salud 1990; 304-310.         [ Links ]

8. Rice M, Valdivia L. A simple guide for design, use, and evaluation of educational materials. Health Educ Q 1991; 18:79-85.         [ Links ]

9. Bernier MJ, Yasko J. Designing and evaluating printed education materials: model and instrument development. Patient Educ Couns 1991; 18:253-263.         [ Links ]

10. Mathis DB. Writing patient education materials. Orthop Nurs 1989; 8:39-42.         [ Links ]

11. Smith H, Gooding S, Brown R, Frew A. Evaluation of readability and accuracy of information leaflets in general practice for patients with asthma. BMJ 1998; 317:264-265.         [ Links ]

12. Slaytor EK, Ward JE. How risks of breast cancer and benefits of screening are communicated to women: analysis of 58 pamphlets. BMJ 1998; 317: 263-264.         [ Links ]

13. Doak LG, Doak CC, Meade CD. Strategies to improve cancer education materials. Patient Educ Couns 1996; 23:1305-1312.         [ Links ]

14. Wells JA. Readability of HIV/AIDS educational materials: the role of the medium of communication, target audience, and producer characteristics. Patient Educ Couns 1994; 24:249-259.         [ Links ]

15. Shepperd S, Charnock D, Gann B. Helping patients access high quality health information. BMJ 1999; 319:764-766.         [ Links ]

16. Svarstad BL, Mount JK. Evaluation of Written Prescription Information Provided in Community Pharmacy, 2001 - Final Report to the U.S. Department of Health and Human Services and the Food and Drug Administration. Available at: URL: http://www.fda.gov/cder/reports/prescriptionInfo/default.htm, accessed on 25/08/2005.         [ Links ]

17. Coulter A, Entwistle V, Gilbert D. Informing Patients: an assessment of the quality of patient information materials. London (UK): King's Fund, 1998. ISBN:1857172140.         [ Links ]

18. Cooper C, Booth K, Fear S, Gill G. Chronic disease patient education: lessons from meta-analyses. Patient Educ Couns 2001; 44:107-117.         [ Links ]

19. Coulter A. Evidence based patient information. BMJ 1998; 317:225-226.         [ Links ]

20. Meade CD, Byrd JC. Patient literacy and the readability of smoking education literature. AJPH 1989; 79:204-206.         [ Links ]

21. Dickinson D, Raynor DK, Duman M. Patient information leaflets for medicines: using consumer testing to determine the most effective design. Patient Educ Couns 2001; 43:147-159.         [ Links ]

22. Williams MV, Baker DW, Parker RM, Nurss JR. Relationship of functional health literacy to patient's knowledge of their chronic disease: a study of patients with hypertension and diabetes. Arch Intern Med 1998; 158:166-172.         [ Links ]

23. Steering Committee for the Collaborative Development of a Long-Range Action Plan for the Provision of Useful Information about Prescription Medicine. Action Plan for the Provision of Useful Information about Prescription Medicine, Unpublished report submitted to The Honorable Donne E. Shalala, Secretary of the U.S. Department of Health and Human Services, December 1996. URL: http://keystone.org/PublicPolicy/FinalActionplan.pdf. accessed on 25/08/2005.         [ Links ]

24. Luz ZMP, Pimenta DN, Rabello A, Schall V. Evaluation of informative materials on leishmaniasis distributed in Brazil: criteria and basis for the production and improvement of health education materials. Cad Saúde Pública 2003; 19:561-569.         [ Links ]

25. Schall VT, Monteiro S, Rebello SM, Torres M. Evaluation of the ZIG-ZAIDS game: an entertaining educational tool for HIV/Aids prevention. Cad Saúde Pública 1999; 15:107-119.         [ Links ]

26. Doak CC, Doak LG, Root JH. Teaching Patients with Low Literacy Skills. 2nd ed. Philadelphia (Pen): J.B. Lippincott Company; 1996. ISBN 0397551614.         [ Links ]

27. Streiner DL, Norman GR. Health Measurement Scales - a practical guide to their development and use. 2ª ed. New York: Oxford; 1995. ISBN:0192626701.         [ Links ]

28. Achutti A, Medeiros AB. Hipertensão Arterial no Rio Grande do Sul. Boletim da Saúde da SSMA-RS. 1985; 12:2-72.         [ Links ]

29. Fuchs FD, Moreira LB, Moraes RS, Bredemeier M, Cardozo SC. Prevalência de hipertensão e fatores associados na região urbana de Porto Alegre. Arq Bras Cardiol. 1994; 63:473-79.         [ Links ]

30. Chobanian AV, Bakris GL, Cushman WC, Green LA, Izzo JL, Jones DL et al. The Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure. JAMA 2003; 289:2560-2572.         [ Links ]

31. Sociedade Brasileira de Hipertensão, Sociedade Brasileira de Cardiologia, Sociedade Brasileira de Nefrologia. III Consenso Brasileiro de Hipertensão Arterial. 1998.         [ Links ]

32. Goldim JR. O consentimento e a adequação de seu uso na pesquisa em seres humanos [dissertation]. Porto Alegre (RS). UFRGS; 1999.         [ Links ]

33. Castro MS, Ferreira MBC. Material Educativo Impresso: influência na adesão ao Tratamento. Rev Bras Farm 1998; 79:45-48.         [ Links ]

34. Rantucci MJ. Pharmacists Talking with Patients: A guide to patient counseling. Baltimore: Williams & Wilkins, 1997. ISBN:0683071270.         [ Links ]

35. Schommer JC, Wiederholt JB. The association of prescription status, patient age, patient gender, and patient question asking about behavior with the content of pharmacist-patient communication. Pharm Res 1997; 14:145-151.         [ Links ]

36. Lewis RK, Lasack NL, Lambert BL, Connor SE. Patient counseling - a focus on maintenance therapy. Am J Health-Syst Pharm 1997; 54: 2084-2098.         [ Links ]

37. Oliveira VZ. Comunicação Médico-Paciente e adesão ao tratamento em adolescentes portadores de doenças orgânicas crônicas [dissertation]. Portoalegre (RS): UFRGS; 2000.         [ Links ]

38. Claesson S, Morrison A, Wertheimer AI, Berger ML. Compliance with prescribed drugs: challenges for the elderly population. Pharm World Sci 1999; 21:256-259.         [ Links ]

39. Mayeaux EJ, Murphy PW, Arnold C, Davis TC, Jackson RH, Sentell T. Improving patient education for patients with low literacy skills. Am Fam Physician 1996; 53:205-211.         [ Links ]

40. Visser A. Education and counseling approaches in chronic conditions. Patient Educ Couns 2000; 41:241-242.         [ Links ]

41. Webber GC. Patient education. Med Care 1990; 28:1089-1103.         [ Links ]

42. Norman SA, Marconi KM, Schelzel GW, Schechter CF, Stolley PD. Beliefs, social normative influences, and compliance with anti-hypertension medication. Am J Prev Med 1985; 1:10-17.         [ Links ]

43. Knight MA. The police surgeon's view: medical paternalism is unacceptable. BMJ 1995; 311:1620-1621.         [ Links ]

44. World Health Organization. Resolution on the role of the pharmacist in support of the WHO revised drug strategy. 47ª World Health Assembly; 1994.WHA47.12        [ Links ]

45. World Health Organization. The role of the pharmacist in the health care system: Report of a WHO Meeting. 1993. WHO/PHARM/94.596.         [ Links ]