SciELO - Scientific Electronic Library Online

 
vol.40 número1Quimioterapia al final de la vida: análisis de la práctica clínica en pacientes adultos onco-hematológicos con cáncerEvaluación de dos sistemas cerrados en el proceso de elaboración de quimioterapia parenteral índice de autoresíndice de assuntospesquisa de artigos
Home Pagelista alfabética de periódicos  

Serviços Personalizados

Journal

Artigo

Indicadores

Links relacionados

  • Em processo de indexaçãoCitado por Google
  • Não possue artigos similaresSimilares em SciELO
  • Em processo de indexaçãoSimilares em Google

Compartilhar


Farmacia Hospitalaria

versão On-line ISSN 2171-8695versão impressa ISSN 1130-6343

Farm Hosp. vol.40 no.1 Toledo Jan./Fev. 2016

https://dx.doi.org/10.7399/fh.2016.40.1.8993 

ORIGINALES

 

Smartphone applications for cancer patients; what we know about them?

Aplicaciones de smartphone para pacientes con cáncer; ¿qué conocemos sobre ellas?

 

 

Roberto Collado-Borrell, Vicente Escudero-Vilaplana, Almudena Ribed-Sánchez, Sara Ibáñez-García, Ana Herranz-Alonso and María Sanjurjo-Sáez

Servicio de Farmacia. Hospital General Universitario Gregorio Marañón Madrid. Spain.

This article has been funded by the Instituto de Salud Carlos III with cofunding by the European Funds. Project Code: PI13/02056.

Correspondence

 

 


ABSTRACT

Background and objective: one of the groups that can benefit most from healthcare applications, are cancer patients. However, not all applications have a sufficient level of evidence. Our objective is to analyze the characteristics of mobile healthcare applications for cancer patients and know the reliability of their information.
Material and methods: a descriptive observational study of mobile apps targeting cancer patients. In November 2014, we searched mobile applications for cancer patients in the App Store (iOS) and Google Play (Android), using the terms "cancer" and "oncology" (English and/or Spanish languages). Applications were downloaded and evaluated. We registered their general characteristics (classification of cancer, last date of actualization, language and others) and their purpose (whether were informative, diagnostic, or preventive purposes) on an Excel® chart. The analysis was completed with an internet search to analyze their scientific evidence.
Results: one hundred and sixty six applications were downloaded. 23.5% were destined for breast cancer. 52.4% upgraded their software in the last year. 98.2 % were in English. Most of the applications had more than one purpose. The most frequent were informative (39.8%), diagnostic (38.6%) and preventive (28.3%). 50.6% presented sufficient scientific evidence.
Conclusions: there are many benefits that are expected from these applications. However, we detected a lack of validity of the information, as well as lack of update of the data. To prevent these apps from becoming a safety problem rather than a useful tool for patients, regulation should be put in place.

Key words: Applications; Smartphone; Cancer; Safe; Effective.


RESUMEN

Fundamento y objetivo: uno de los grupos que más se pueden beneficiar de las aplicaciones en salud son los pacientes con cáncer. Sin embargo, no todas tienen un nivel de evidencia suficiente. El objetivo es analizar las características de las aplicaciones móviles destinadas a los pacientes con cáncer y conocer la fiabilidad de su información.
Material y métodos: estudio observacional descriptivo de las aplicaciones móviles destinadas a los pacientes con cáncer. En noviembre de 2014 se realizó una búsqueda en los sistemas operativos App Store (iOS) y Google Play (Android), utilizando los términos "cancer" y "oncology" (idiomas inglés y/o español), se descargaron y evaluaron. En una tabla Excel® se registraron sus características generales (tipo de cáncer, fecha de última actualización e idioma, entre otras) y finalidad (informativa, diagnóstica y preventiva, entre otras). El análisis se completó con una búsqueda en internet para analizar su evidencia científica.
Resultados: se descargaron 166 aplicaciones. El 23,5% estaban destinadas al cáncer de mama. El 52,4% actualizaron el software en el último año. El 98,2% estaban en inglés. La mayoría tenían más de una finalidad. Las más frecuentes fueron: informativa (39,8%), diagnóstica (38,6%) y preventiva (28,3%). El 50,6% presentaban una evidencia científica suficiente.
Conclusiones: son muchos los beneficios que se esperan de estas aplicaciones. Sin embargo, se ha detectado una falta de validez de la información, así como falta de actualización de los datos. Para evitar que se conviertan en un problema de seguridad en lugar de una ayuda para el paciente, es necesaria su regulación.

Palabras clave: Aplicaciones; Smartphone; Cáncer; Seguro; Efectivo.


 

Introduction

In recent years we have witnessed an increasing development in new technologies; mobile phones, and particularly Smartphones, have become an essential tool in daily life1-5. According to the National Observatory for Telecommunication and Information Society, it is estimated that 53.7% of the > 15-year-old Spanish population have a Smartphone, and 39.6% of the overall population have accessed internet through their mobile phone as an information source during the last three months of 20136. The health area has also experienced an important change in the way in which information is accessed. Internet has become a tool used for reference both by healthcare professionals and patients7. The concept of mHealth (mobile health) has appeared within this context. The WHO Global Observatory for Electronic Health defines mHealth as those medical and public health practices supported by mobile devices8.

Applications (apps) are one of the main tools in Smartphones7,9, and those devoted to health promotion and self-care are considered highly useful1,9. Those mHealth apps for patients were created with the objective of becoming a guide for disease management, and to allow remote monitoring by healthcare professionals10. However, these apps can be targeted to a very heterogeneous population. One of the groups that can benefit most from these apps are cancer patients. Continuous developments in this area are leading to higher survival rates, and cancer patients have become chronic patients1. Thus, on 2012 in Spain, the prevalence rate at 5 years was 1,467.6 cases per each 100,000 inhabitants11. On the other hand, the majority of the new drugs developed have oral administration, and even though they provide higher safety, autonomy, and quality of life , these are considered high-risk medications due to the potential mistakes associated with their use (dosing, lack of treatment compliance, etc.)12. Besides, due to the inherent characteristics of the oral way of administration, there has been an increase in the responsibility of patients regarding their therapy, because they have become ultimately responsible for its administration. All this, associated with therapy complexity and the numerous questions faced by cancer patients during their treatment, has led to an increasing search for information on-line and in apps1.

On the other hand, the fast technological advance and development of these apps leads us to these questions: Are these apps a truthful and confirmed source of information? Are these apps reliable? Not all of them have been prepared by healthcare professionals, which generates a certain level of uncertainty about their contents and the risk entailed for patients2,7,9,13,14. Besides, due to the high variety of apps available, it is difficult to know which will be most adequate for each patient, and the best way to use it so that it will really be effective10. Healthcare professionals could play an essential role not only for the review or verification of the contents of these apps, but also for their prescription to the most adequate patient, and the recommendation of those most reliable.

For all the above, the objective of this study is to analyze the characteristics of Smartphone applications for cancer patients. There was an assessment of their objective, year of update, type of cancer, cost, and their developers.

 

Materials and Methods

An observational descriptive study of all Smartphone applications associated with cancer and targeted to patients and/or their relatives or carers.

On November, 2014, a search was conducted in the App Store (iOS) and Google Play (Android) operating systems, within the categories "medicine" and "health and physical fitness" or "wellbeing": the terms "cancer" and "oncology" were used for the search in the main search engine. Only those apps for patients and/or relatives and carers were selected, and only those in English and/or Spanish. All apps targeted at smoking cessation were excluded, as well as those with no specific target patient.

On a first step, all available information within the platform was analyzed, and only those apps which met the criteria previously indicated were downloaded and evaluated, regardless of cost. The iOS applications were downloaded to an iphone 4s, and Smartphone apps were downloaded to a Samsung Galaxy S2.

An Excel® chart was designed for data extraction, where the general characteristics of these apps were registered: name, operative system, date of the last update, cost, category, language, and target type of cancer. Any piece of data which could provide information about their quality was also included.

Subsequently, their contents were classified based on their objective, within the following categories: overall information, diagnosis, prevention, calendar, treatment registration, disease evolution status, symptoms for disease follow-up, and others. The analysis was completed with a search in the developer web pages, in order to analyze their contents. It was considered that qualified professionals had been involved in the apps contents if these had been developed by healthcare organizations: scientific foundations and societies, universities, healthcare administrations and hospitals.

Data were analyzed by two independent researchers with experience in the oncohematology area, as well as in healthcare technology management. Data analysis was conducted using the SPSS® statistical program, version 18.0, through descriptive statistics. Cohen's Kappa test was conducted in order to guarantee the reliability of those data analyzed by 2 independent observers.

 

Results

After analyzing the information available in the platform, 4 apps were excluded due to lack of knowledge of their target patient; and 166 apps in total were downloaded, associated with cancer and targeted to patients. Out of these, 75 were available in Android, 59 in iOS, and 32 in both platforms. The initial characteristics of the apps analyzed appear in table 1.

 

 

Characteristics of apps:

Out of all the 166 apps, 84 (50.6%) were downloaded within the Medicine category. Regarding their cost, only 20 required payment, with a mean cost of 2.15€ (SD: 1.73€). In terms of type of cancer, the apps were targeted at overall cancer, or for different tumours specifically. Based on type of cancer, the majority were for breast cancer (23.5%), followed by skin cancer (9%). Table 2 shows the allocation of apps based on type of cancer. Regarding their date of update, 52.4% had updated their software within the past year. In terms of language, 98.2% of the apps downloaded were in English, 1.8% in Spanish, and 15.1% in both languages.

 

 

Contents of the apps:

The majority of apps had more than one purpose. The most frequent were informative (39.8%), diagnostic (38.6%) and preventive (28.3%).

- Out of those 66 (39.8%) apps which provided overall information about cancer, among other purposes, 42 (63.6%) were centred in general information about antineoplastic agents.

- Out of those apps focused on diagnosis, 20 (31.3%) were targeted to breast cancer patients, and 14 (21.9%) to skin cancer. Of those targeted to breast cancer patients, 12 (18.8%) informed about how to conduct a breast examination, or included reminders to conduct them. The apps focused on skin cancer included images or tools to capture photos allowing to analyze pigmented skin lesions.

- 47 (28.3%) apps were targeted to cancer prevention, through education and information, by promoting a change of lifestyle through physical exercise, diet modification, and other basic measures. Out of these, 21 (44.7%) were targeted to cancer in general, and only 2 (4.3%) were focused on skin cancer.

- Regarding the rest of purposes, 16 (9.6%) apps acted like calendars to provide information associated with the management of tests, appointments, treatment schedule, etc. 10 (6%) were targeted to registration and follow-up of treatments and adverse effects, in order to allow a close follow-up of said treatments. In terms of other purposes, 9 (5.4%) apps were targeted to emotional support for patients, through personal experiences, music, etc. A lower proportion of apps, 8 (4.8%), were dictionaries for cancer terms. Regarding the estimation of the risk of cancer, 4 (2.4%) apps were found for breast, lung, prostate, and overall cancer; through personalized information, it was possible to calculate the risk of developing cancer in the future.

A 48.8% of the apps had been developed by healthcare organizations. In terms of their developer, 38.6% were scientific foundations, 10.8% had been developed by scientific societies, 10.8% by hospitals, and 7.2% by the pharmaceutical companies, among others.

The Cohen's Kappa Test used to guarantee the results of the analyzed data had a score of 1, with a 100% level of concordance.

 

Discussion

Smartphones, and specifically apps, have represented a major advance in the area of Medicine in recent years, with a great impact on search methods and access to information7,15. There are currently over 40,000 mHealth apps available in the "app store" and their use in clinical practice is increasing, particularly those apps targeted to healthcare2,9. Thanks to a study by the Institute for Healthcare Informatics (IMS health) consultancy firm, we know that only 16,000 of all these apps had a certain value for being used in health-related topics8. These mHealth apps have been the object of studies because of their potential application in numerous medical areas, including cancer10,15,16. However, there is limited evidence from studies analyzing the contents of apps targeted at cancer patients3.

The present study, where 166 apps targeted to cancer patients have been analyzed, highlights the high number of apps designed for breast cancer patients, which represent almost a fourth of the total. On the other hand, prostate and lung cancer represented 3.6% each and 0.6% are for colorectal cancer. However, these data are not representative of the real population. In a report prepared by Globocan, the most frequent types of cancer in Spain on 2012, according to total number of cases, were breast and colorectal in women, prostate and lung in men, and colorectal and prostate in both genders17.

The level of update of these apps is undoubtedly one of the factors which might create more concern. Apps are created, but no maintenance is conducted in a major proportion of them. In our study, we have found that only half of the apps had been updated during the past year, while in 30% the last update had been conducted over 2 years ago. A condition in constant evolution like cancer requires continuous updates. Only in Spain, 9 new treatments were authorized during 2012 in the area of oncohematology18. We have also found apps about breast cancer with obsolete contents, regardless of being validated by medical professionals. As an example, we can highlight that in some of them there is no mention of the new classifications based on the HER-2 receptor, and not even of hormonal receptors.

Regarding language, the low number of apps in Spanish is noticeable. Only 15.1% of apps were in Spanish. This could mean an access problem for the almost 500 million of Spanish-speaking people throughout the world19.

In this study, it was observed that the majority of apps were targeted to increasing the overall awareness of cancer. Only 26 (15,7%) apps provided individual self-monitoring tools, either through diaries or records for follow-up of adverse effects, treatment, etc. For some years now, self-monitoring tools using mobile phones have demonstrated efficacy in different conditions1. Weaver et al. showed that a telemedicine system recording adverse effects through a mobile phone, and providing advice about toxicity management, was useful to increase patient safety20. A bibliographic review demonstrated that interventions through mobile phones in terms of reminders and text messages can help to improve health out-comes21. Thanks to telemedicine, it has been observed that there is a higher involvement of patients through higher attendance to doctor appointments and a better physician-patient communication. Benefits in health have also been observed, through changes in behaviour, better treatment compliance, and an improvement in disease control1,9,22,23. However, in a review by Bender et al., it was observed that only 17.2% of apps provided tools for self-care, prevention or detection, which is a rate very similar to the one observed in our study (1).

Finally, our study shows an alarming lack of involvement by qualified professionals in the contents of apps, because only 48.8% had been developed by healthcare organizations (scientific foundations and societies, universities, healthcare administrations and hospitals), which could entail a risk for patient safety. Only half of the apps could be considered to be supported by a scientific basis. Over a third part of these were associated with scientific foundations, and in a lower rate with hospitals and scientific societies. These results obtained were consistent with the bibliography available1,4. Pandey et al., in an analysis of apps associated with cancer, observed that only 55.8% of these apps provided scientifically validated data, and there was a difference in scientific validity between those targeted at healthcare professionals and those targeted to patients7. Various studies have warned about the lack of professional involvement and evidence in the development of apps, increasing the concern about their contents, and potentially endangering patient safety1,2,15. Wolf et al., in a study about apps for melanoma detection, found in 3 of each 4 apps a wrong classification for 30% of melanomas. These apps, not subject to medical supervision, could lead to a delay in melanoma diagnosis24.

All the potential that apps might have for healthcare is set against the lack of involvement by qualified professionals. Just like medications and healthcare products have a regulatory setting, there should be legislation for health apps. The FDA, a pioneer in apps regulation, prepared guidelines in 2013 with recommendations to guarantee their quality25. On the other hand, there is no framework in Spain to regulate the contents of these apps. However, the European 93/42/CEE rule regarding healthcare products determines that computer programs designed by their manufacturer for specific objectives of diagnosis and/or therapy, and which are involved in their good performance, targeted by their manufacturer for use in human beings with objectives of diagnosis, prevention, monitoring, treatment, or relief of a disease, should be regarded as healthcare products26. Therefore, according to this rule, an app meeting these criteria could be considered a Type I healthcare product. In this case, the manufacturers should declare the CE mark without any intervention by a reported agency. The European Commission prepared the MEDDEV 2.1/6 guidelines, which help developers to determine through an algorithm if said ruling is applicable to their apps27. For all the above, an app would deserve to be considered a healthcare product if clinical information was processed through algorithms.

Even though in Spain there is no regulatory framework as such, similar initiatives have been conducted in the regions of Andalusia and Catalonia. Besides, some certifying entities grant seals of quality, such as the British NHS (National Health Service)28 and the "Calidad App Salud" (Health App Quality) by the Healthcare Quality Agency of Andalusia29. In order to be considered safe and high quality, according to the Healthcare Quality Agency of Andalusia, any app must follow a series of recommendations focused on the following aspects: design and relevance, quality and safety of information, provision of services, and confidentiality and privacy, all of them developed through 31 requirements29. In our study, we have found that only 1 of the apps analyzed had this seal of quality. However, in spite of all these initiatives, there is currently no official national regulation to certify apps, which might create certain confusion among their target population.

In order to guarantee the reliability of the data analyzed, and because our study is observational, we have conducted a Cohen's Kappa test, which has obtained 100% reliability among the observations by our 2 independent researchers, providing robustness to our outcomes, which coincide to a high extent with those reviews published30.

Among the limitations of the present study, we can highlight the source of our search. For this study, we analyzed the apps available in App Store and in Google Play, platforms which represent approximately 85% of the total market share2. However, there are other platforms such as Windows Phone or Blackberry, which might have other apps for cancer patients. Moreover, the transversal nature of the study reflects the current moment, and therefore this information can evolve over time.

There are many benefits expected from these apps, but that is the reason why we must be particularly careful with their design. Regulation is required in order to prevent these tools from becoming a safety problem instead of an aid for cancer patients. Therefore, this is a time when technology is in full swing, which can entail a huge benefit for patients if we know how to take advantage of it. The number of mobile applications targeted to cancer patients has doubled during the past year. The objective of these apps is to promote a change in patient behaviour, to control and get to know their symptoms, to encourage an early diagnosis, and to act as a source of information about these conditions. However, as described in this study, we have detected a major number of applications that have not been developed by qualified professionals, as well as some lack of data update which can trigger a great safety problem. Therefore, once there is specific regulation about health apps, we will be closer to an individualized prescription of said apps to patients as part of their treatment.

 

Bibliography

1. Bender JL, Yue RYK, To MJ, Deacken L, Jadad AR. A Lot of Action, But Not in the Right Direction: Systematic Review and Content Analysis of Smartphone Applications for the Prevention, Detection, and Management of Cancer. J Med Internet Res. 2013 Dec 23;15(12):e287.         [ Links ]

2. Cantudo Cuenca MR, Cantudo Cuenca MD, Morillo Verdugo R. Availability and medical professional involvement in mobile healthcare applications related to pathophysiology and pharmacotherapy of HIV/AIDS. Eur J Hosp Pharm. 2013 Dec 1;20(6):356-61.         [ Links ]

3. Mirkovic J, Kaufman DR, Ruland CM. Supporting Cancer Patients in Illness Management: Usability Evaluation of a Mobile App. JMIR MHealth UHealth. 2014 Aug 13;2(3):e33.         [ Links ]

4. Boulos M, Wheeler S, Tavares C, Jones R. How smartphones are changing the face of mobile and participatory healthcare: an overview, with example from eCAALYX. Biomed Eng OnLine. 2011;10(1):24.         [ Links ]

5. Mosa AS, Yoo I, Sheets L. A Systematic Review of Healthcare Applications for Smartphones. BMC Med Inform Decis Mak. 2012;12(1):67.         [ Links ]

6. XLII Oleada del Panel Hogares "Las TIC en los hogares españoles" (4T/2013) (consultado 08 Dic 2014). Disponible en: http://www.ontsi.red.es/ontsi/sites/default/files/xlii_oleada_las_tic_en_los_hogares_espanoles_4t_2013.pdf.         [ Links ]

7. Pandey A, Hasan S, Dubey D, Sarangi S. Smartphone Apps as a Source of Cancer Information: Changing Trends in Health Information-Seeking Behavior. J Cancer Educ. 2013 Mar;28(1):138-42.         [ Links ]

8. IMS Institute for Healthcare Informatics. Patient Apps for Improved Healthcare from Novelty to Mainstream (consultado 08 Dic 2014). Disponible en: www.imshealth.com.         [ Links ]

9. Van Velsen L, Beaujean DJ, van Gemert-Pijnen JE. Why mobile health app overload drives us crazy, and how to restore the sanity. BMC Med Inform Decis Mak. 2013;13(1):23.         [ Links ]

10. San Mauro Martín I. Aplicaciones Móviles En Nutrición, Dietética Y Hábitos Saludables: Nutr Hosp. 2014 Jul 1;(1):15-24.         [ Links ]

11. Las Cifras del Cáncer en España 2014 (consultado 08 Dic 2014). Disponible en: http://www.seom.org/seomcms/images/stories/recursos/Las_cifras_del_cancer_2014.pdf.         [ Links ]

12. Institute for Safe Medications Practices. ISMP's list of high-alert medications. Huntingdon Valley (PA): ISMP; 2007.         [ Links ]

13. Rozati H, Shah SP, Shah N. Smartphone Applications for the Clinical Oncologist in UK Practice. J Cancer Educ. 2014.         [ Links ]

14. Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling Up mHealth: Where Is the Evidence? PLoS Med. 2013 Feb 12;10(2):e1001382.         [ Links ]

15. Cantudo-Cuenca MR, Robustillo-Cortés MA, Cantudo-Cuenca MD, Morillo-Verdugo R. A better regulation is required in viral hepatitis smartphone applications. Farm Hosp. 2014 Apr;38(2):112-7.         [ Links ]

16. Robson Y, Blackford S, Roberts D. Caution in melanoma risk analysis with smartphone application technology: Correspondence. Br J Dermatol. 2012 Sep;167(3):703-4.         [ Links ]

17. Ferlay J, Soerjomataram I, Ervik M, Dikshit R, Eser S, Mathers C, Rebelo M, Parkin DM, Forman D, Bray, F. GLOBOCAN 2012 v1.0, Cancer Incidence and Mortality Worldwide: IARC.         [ Links ]

18. AEMPS. Ficha de los medicamentos autorizados por la aemps en el año 2012 (consultado 21 Dic 2014). Disponible en: http://www.aemps.gob.es/informa/boletinMensual/2012/docs/ficha-nuevos-medicamentos-autorizados-AEMPS-2012.pdf.         [ Links ]

19. El español: una lengua viva. Informe 2014. Instituto Cervantes (consultado 09 Ene 2015). Disponible en: http://eldiae.es/wp-content/uploads/2014/07/El-español-lengua-viva-2014.pdf.         [ Links ]

20. Weaver A, Young AM, Rowntree J, Townsend N, Pearson S, Smith J, et al. Application of mobile phone technology for managing chemotherapy-associated side-effects. Ann Oncol. 2007 Nov;18(11):1887-92.         [ Links ]

21. De Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, Car J, Atun R. Mobile phone messaging telemedicine for facilitating self management of long-term illnesses. Cochrane Database of Syst Rev 2012 Dec 12.         [ Links ]

22. Mira JJ, Navarro I, Botella F, Borrás F, Nuño-Solinís R, Orozco D, et al. A Spanish Pillbox App for Elderly Patients Taking Multiple Medications: Randomized Controlled Trial. J Med Internet Res. 2014 Apr 4;16(4):e99.         [ Links ]

23. Klasnja P, Hartzler A, Powell C, Pratt W. Supporting cancer patients' unanchored health information management with mobile technology. AMIA Annu Symp Proc. 2011;2011:732-41.         [ Links ]

24. Wolf JA, Moreau JF, Akilov O, Patton T, English JC, Ho J, et al. Diagnostic Inaccuracy of Smartphone Applications for Melanoma Detection. JAMA Dermatol. 2013 Apr 1;149(4):422.         [ Links ]

25. U.S. Food and Drug Administration. Mobile medical applications: guidance for industry and food and drug administration staff. (consultado 09 Ene 2015). Disponible en: http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM263366.pdf.         [ Links ]

26. Directiva 93/42/CEE del Consejo de 14 de junio de 1993 relativa a los productos sanitarios (consultado 09 Ene 2015). Disponible en: http://www.tecnologias-sanitarias.com/MD/93-42-EEC-esp.pdf.         [ Links ]

27. Guidelines on the qualification and classification of stand alone software used in healthcare within the regulatory framework of medical devices. MEDDEV 2.1/6 (consultado 09 Ene 2015). Disponible en: http://ec.europa.eu/health/medical-devices/files/meddev/2_1_6_ol_en.pdf.         [ Links ]

28. NHS choices Health Apps Library (consultado 09 Ene 2015). Disponible en. Available from: http://apps.nhs.uk/.         [ Links ]

29. Estrategia de calidad y seguridad en aplicaciones móviles de salud. (consultado 11 Ene 2015). Disponible en. Available from: http://www.calidadappsalud.com/.         [ Links ]

30. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar;33(1):159-74.         [ Links ]

 

 

Correspondence:
Correo electrónico: vicente.escudero@salud.madrid.org
(Vicente Escudero-Vilaplana).

Recibido: el 25 de marzo de 2015;
Aceptado: el 12 de octubre de 2015.

Creative Commons License Todo o conteúdo deste periódico, exceto onde está identificado, está licenciado sob uma Licença Creative Commons