Keywords: Pruebas genéticas. Metodología. Sensibilidad y especificidad.

The advances derived from the Human Genome Project have led to the development of new kinds of diagnostic tests, genetic, molecular or proteomic, which can be incorporated into clinical practice^1-4. In Spain currently there are tests based on DNA analysis for the diagnosis or prognosis of 214 genetic diseases⁵, and it is foreseen that in the coming years this number of tests will multiply. This has given rise to the elaboration of evaluation frameworks for the incorporation of new genetic tests to the list of services of health systems⁶. The appearance of these new tests creates expectations, which often are not corroborated in clinical practice. This is due to the fact that sometimes the results of basic research are publicised without awaiting confirmation from the results of clinical research; at other times the clinical validation of the tests lacks methodological rigour, precisely for their having been developed in basic research environments without the collaboration of investigators with experience in clinical and epidemiological research. In any case the new genetic-molecular technologies should not be introduced for a specific clinical task (screening and genetic counsel, diagnosis, risk evaluation, etc.), without prior examination of both their validity for this specific purpose as well as their effects in health terms¹.

A methodological deficit in the quality of research in other diagnostic areas has been described^7-12. In genetic-molecular research, however, a single work analysed the methodology of the studies published in four international medical journals, showing that 63% of them fulfilled only one of the required methodological criteria¹³.

In order to improve the articles on diagnostic research and, as a result, the scientific quality of this research, the STARD (Standards for Reporting of Diagnostic Accuracy)¹⁴ was initiated, and its recommendations have been incorporated by most of the scientific journals. Moreover, Little and collaborators¹⁵ presented a list of standards shared in part by STARD, but with some aimed solely at research into markers based on genotypes, which proposed to serve as a guide for authors as well as editors and reviewers of genetics articles.

All original articles studying the diagnostic accuracy of genetic, molecular and proteomic tests from January 2002 to June 2005 in four international clinical journals (JAMA, British Medical Journal, Lancet and New England Journal of Medicine) and 2 journals centred on the field of cancer research belonging to the American Association for Cancer Research (AACR) (Cancer Research and Clinical Cancer Research) were reviewed.

Selection of articles

A search was conducted through the Medline computerised bibliographic system using the search strategy employed by Devillé and collaborators¹⁶, combining the MESH terms «sensitivity and specificity» with the words «false negative» and «accuracy». In order to improve the sensitivity of the search it was widened with the MESH term «area under the curve» and the words «diagnostic odds ratio» and «likelihood ratio». After reading the summaries, all the articles in which human genetic material was analysed at molecular level and all proteomic studies were selected, without limiting the search to any particular laboratory technique or clinical condition.

Methodological standards

24 of the 25 criteria belonging to the STARD¹⁴ guide were applied (the first standard of this guide was omitted for being the criterion for inclusion in our study). The given definitions were followed in all of them. Although they can be consulted in detail in the reference publications, the following shows the methodological standards grouped into four blocks:

1. Introduction. Within the study aims, an estimate of the diagnostic accuracy of a test should be included, or the comparison of that accuracy among the tests or participating groups (1).

2. Methods. The inclusion and exclusion criteria of the participants in the study should appear (2) and the place where the data or samples were gathered, as well as specification of the reason for their choice (3). In the case of it not being a consecutive sample of patients then how they were assembled should be explained (4), as well as the directionality of the study (5). On the test being evaluated, the reference standard used should be defined (6) and technical specifications given, which include how and when the measurements were made, for both the diagnostic test and the reference standard (7). The units and/or categories of the results of the diagnostic test and the reference standard should also be described (8). Additionally, the number, training and experience of the people who have carried out and interpreted the tests and the reference standard should be specified (9), and if the results have undergone blind reviewing (10). The statistical methods used to calculate and compare measurements of diagnostic accuracy and to quantify the standard error should be described (11), as too should the methods for calculating the reproducibility of the tests (12).

3. Results. This section compiles characteristics of when the study was carried out (13), the clinical and demographic characteristics of the study population (14) and the number of participants which satisfied the selection criteria, and which then received or not the diagnostic test and/or the reference standard (this can be illustrated with a flow diagram) (15). Also included is the time interval between the application of the diagnostic test, the reference standard and any other treatment administered between these (16), the distribution of the severity of the disease among the patients (17), a table with the results of the diagnostic tests in comparison with those obtained from the reference standard (18) and the communication of any adverse effect produced during the study (19). Lastly, the diagnostic accuracy and the statistical precision should be estimated (20), how indeterminate results have been treated (21), the estimation of the variability of the diagnostic accuracy for each subgroup of participants, researchers or centres (22), and the calculation of the reproducibility of the technique (23).

4. Discussion. In this section it is important to comment on the clinical applicability of the study findings (24).

Observer variability

In order to calculate the inter-observer variability in the application of the criteria, 4 of the 44 works were selected at random and reviewed independently by the two observers. The degree of agreement reached between them was 86% (confidence interval [CI] 95%, 78-92).

The creation and management of the data base and the statistical analyses of the results were performed with the SPSS statistical package (version 12 for Windows Inc., Chicago, Illinois).

Mean fulfilment of the 24 methodological standards applied was 9.8 (CI 95%, 8.8-10.6), and none of the 44 articles evaluated satisfied more than 18 criteria. No statistically significant differences were observed in the fulfilment of the criteria before and after the year 2003 (publication of the STARD guide¹⁴).

Analysing the criteria individually, the greatest deficiencies were found in the material and methods section (table 2). Only 11 works (24%) specified the determination of this diagnostic accuracy as being among their objectives. Also low was the frequency with which inclusion and exclusion criteria and the place of origin of the subjects was specified (23%); the way in which these were selected (20%), or whether this had been planned before or after the performance of the diagnostic test (27%). Regarding the performance of the test, there was a low level of fulfilment in evaluators description, 1 work (2%), and whether the interpretation of the results was blinded, 12 articles (27%). The description of the methods for studying the reproducibility of the test was only communicated in 10 studies (23%) and the results obtained being shown in only 6 articles (13%).

With regard to the presentation of the results (table 3), most defined the clinical and demographic characteristics of the study population, 34 (77%), but the reasons for which some patients who fulfilled the inclusion criteria but then did not continue in the study are hardly given, 11 (25%). Only 12 studies (27%) investigated whether the diagnostic sensitivity and specificity varied according to relevant clinical subgroups. Communication and analysis of the indeterminate or imprecise results was very infrequent, 5 (11%).

The methodological quality found in both the clinical journals as well as those dedicated to cancer research is similar, although with some specificity. The most analytical aspects, like the specification of the technical characteristics, are given more attention to in the cancer journals; while others more related to the way in which to carry out the study, as for example the blind reviewing of the results obtained, are better fulfilled in the clinical journals.

The only antecedent in the field of genetic-molecular diagnosis of this kind of investigation was a review carried out in 1995 on the same four clinical journals as those analysed in this work¹³. Although the same methodological criteria were not used, those employed at that time are included in the STARD guide, and are therefore comparable. As in this review, no work fulfilled an acceptable number of methodological standards, the calculation of the genetic test's reproducibility was unusual and there was a general lack of blind reviewing in the interpretation of the results. Hence it would appear that genetic-molecular diagnostic studies are more impermeable to clinical-epidemiological advances.

Our intention of facilitating comparison with previous works led us to the application of the same restrictive selection criteria as employed previously^7,9-12, and as a result to a reduced sample of articles, a fact which limits the reach of the results. Nevertheless, by including all the original articles which have been published on the field of genetic and molecular diagnosis in the best international journals of clinical medicine and in the two publications of the American Association of Cancer Research (for containing an important part of the works published on the same matter), an adequate perspective of the quality of research and publication of this kind of study is offered. The guide employed for the evaluation of the articles (STARD) is very recent but has been rapidly disseminated among most journals which publish diagnostic evaluations; in fact, two of them (JAMA and Lancet) have included it among the rules for authors^17,18, although not so New England Journal of Medicine or the two cancer journals. It could be argued that there is little experience in its application, and it is true that there may be doubts about the applicability of some of its criteria due to the lack of precise indications. However, available experience shows that concordance on its application is high among observers^11,12, as it has been too in the sample carried out in the present study. Moreover, many of the STARD criteria have already been applied in previous methodological guides^7-12, which has facilitated its use. Studies on diagnosis in genomics, proteomics and related fields have some specificities in their validation¹⁹ which the STARD guide does not include, and although various initiatives have originated for the elaboration of a set of guidelines suitable for the validation of these tests¹⁵, no definitive one has been drawn up yet. This fact represents a limitation in that some intrinsic problems have not been evaluated, although they are not exclusive to this field of research. One example is «over-fitting», which occurs for example when a proteomics profile is proposed and selected from a broader group after countless discrimination trials, which affects external validity. This problem is controlled by determining the reproducibility of the discriminatory pattern observed in samples independent of the original. If the discrimination pattern cannot be reproduced in other samples, the existence of «overfitting» is probable²⁰.

Another limitation to bear in mind is the study period, since the STARD criteria were first published early in the year 2003¹⁴, and this study recompiles articles from 2002 to 2005. Certainly it is early for the initiative to have had effect, at least in the journals which have included it in the instructions for their authors; and our study does not have the statistical capacity to establish comparisons. However, the fact that it was not published does not explain the low quality observed, as many of its criteria are essential requirements in most observational studies already partially compiled --the most relevant ones-- in other guides and previous publications.

Not all the requirements have the same importance in terms of their effects on the validity of the evaluated studies. Bearing in mind the main characteristics that a genetic test should possess for its correct use²¹, it is important to underline the key aspects of validity which have shown themselves to be the most deficient. In first place are the analytical characteristics of the test evaluated, as due to the current rapid development of genetic tests for the same test there may be different techniques and one should know which of them shows the most acceptable values of diagnostic accuracy. Hence, the scanty calculation of the technique's reproducibility in the investigations evaluated is notable. In previous reviews this concept was already considered essential^7-12, and yet in the field of genetics or proteomics, where the value of the determinations will also depend on the capacity to give the same result when applied to the same patients under the same conditions¹⁹, it has been neglected all too often.

Secondly, another key methodological deficiency, and one which limits the applicability of the results, is the inadequate description of the inclusion and exclusion criteria of the subjects participating in the study, these barely appearing in less than half of the works evaluated. This lack of information impedes detailed consideration of the clinical applicability of the test evaluated, which is an indispensable element in this kind of investigation²¹.

To summarize, the future projection of the new genetic-molecular technologies in the field of diagnosis and precocious detection is very limited by the lack of rigour in the research on these tests. These problems are at the root of the reiterated false expectations which are created around these diagnostic tests. If an effective interdisciplinary relationship (basic, clinical, epidemiological) which facilitated the rigorous development of these genetic-molecular diagnostic tests existed then a lot of unnecessary research would be avoided. Furthermore, adherence to the methodological standards would allow the genetic-molecular and proteomic tests to become very useful tools in clinical and health service research, especially in the field of cancer²².

Acknowledgements

This work was financed by ISCIII, Red de Centros RCESP C03/09 and the Ministerio de Sanidad y Consumo, Instituto de Salud Carlos III (grants for post-training health specialist contracts).

Address for correspondence:
Dra. Blanca Lumbreras.
Departamento de Salud Pública, Historia de la Ciencia y Ginecología.
Campus de San Juan. Facultad de Medicina.
Universidad Miguel Hernández.
Ctra. de Valencia, km 8,7.
03550 San Juan de Alicante. España.
Correo electrónico: blumbreras@umh.es

Recibido: 4 de noviembre de 2005.
Aceptado: 12 de enero de 2006.