Participants (Units of Analysis)

Following PRISMA recommendations (^{Page et al., 2021}), the inclusion criteria for the units of analysis (articles) in this study were as follows. All publications a) relied on observational methodology; b) were empirical; c) presented the usual sections of such studies, e.g., introduction, method, results, and discussion; and d) were written in English or Spanish.

The paper selection was based on an exhaustive search in PsycINFO, SCOPUS, Web of Science, SportDiscus, PSICODOC and Google Scholar to search title, abstract, keywords and full text for the term “observational methodology,” with December 1, 2022, as the cut-off date. The reference list from the articles collected in this step were also examined to identify additional studies.

Instruments

Primary papers included were coded using the MQCOM (^{Chacón-Moscoso et al., 2019}), which assesses the methodological quality in observational methodology studies. It presented validity evidence based on the content of the items and adequate intercoder reliability (> .75). The MQCOM is comprised of 16 rating scale items. Each item receives a score for methodological quality levels of 0 (low), .5 (medium), or 1 (high). Exceptionally, item 11 (Data analysis) had four possible scores (from the lowest to the highest methodological quality level, 0, .33, .67 and 1). This item had a different response scale than the others because it was based on a previous checklist with a rigorous content validity and intercoder reliability study where experts highly recommended increasing the graduation of this item (^{Chacón-Moscoso et al., 2019}). Nonetheless, this item maintains the monotonic incremental function of all ordinal items of the scale, with a range of 0-1 prior to the statistical analysis. The coding manual is available at https://osf.io/uv7cj.

Procedure

Mendeley Reference Manager was used for the search for papers to organize and handle the information obtained through the literature search. During the first screening, the inclusion criteria were applied to the title, keywords, and abstract. The resulting studies were assessed in a second stage in which the inclusion criteria were applied to the full texts. Two coders (DLA and IFM) applied the criteria independently. In case of disagreements, a third coder (SCM) mediated until an agreement was reached.

For the data extraction, IFM and DLA were trained to apply the MQCOM. Each item and its response options were explained. MTA mediated if the explanations differed. Then both coders independently applied the scale to two observational methodology papers to compare the coding. In case of disagreements, a third coder (SSC) mediated. After the training, the coders applied the scale independently to a randomly selected 25% of the sample. Finally, once a high level of consensus was achieved (> .7), DLA applied the scale to the full sample. The data extraction database is available at https://osf.io/m6pvh .

Data Analysis

Using SPSS 27.0, the Intraclass Correlation Coefficient (ICC) was calculated to evaluate both inter and intracoder concordance. Values higher than .7 were considered indicators of adequate reliability (^{Portney & Watkins, 2000}). Additionally, descriptive statistics were provided.

A parallel analysis was then conducted using FACTOR 12 (^{Ferrando & Lorenzo-Seva, 2017}) to obtain empirical evidence of dimensionality. Values higher than .95 for Unidimensional Congruence (UniCo) and higher than .85 for Explained Common Variance (ECV) suggests that data can be treated as essentially unidimensional (^{Ferrando & Lorenzo-Seva, 2018}).

After the parallel analysis, the database was randomly split into two subsamples of 325 papers. With one of the subsamples, an exploratory factor analysis was carried out using SPSS 27.0. First, a polychoric correlation matrix was created with all the variables included in the analysis. Following this step, the statistical assumptions of the matrix were checked by running Bartlett’s test of sphericity, where significant results were considered acceptable, and the Kaiser-Meyer-Olkin (KMO) test, where values higher than .5 were considered adequate (^{Bartlett, 1954}; ^{Kaiser & Rice, 1974}). Finally, the exploratory factor analysis was done using principal axis method of factor extraction and Kaiser’s varimax criterion for orthogonal rotation (^{Field, 2018}).

After the exploratory factor analysis, a confirmatory factor analysis was carried out with the second subsample. SPSS 27.0 was used to calculate the internal consistency of the test and the average discrimination index. The internal consistency of the items was measured using Cronbach’s α, where values higher than .7 were considered appropriate. For the average discrimination index, values higher than .4 were considered excellent, .3 - .39 good, and .2 - .29 acceptable (^{Barbero-García et al., 2015}).

The bivariate normality assumption was checked with PRELIS 12 to confirm the suitability of polychoric correlations (^{Holgado-Tello et al., 2010}). The chi-square test (χ²) was run between each pair of correlations assuming a 95% confidence interval with Bonferroni correction α/c (95% confidence level α = .05, and number of contrasts c = [number of items x number of items - 1]/2). The percentage of acceptance of the bivariate normality assumption was then calculated. Additionally, to overcome the large sample size bias in the chi-square test, the Root Mean Square Error of Approximation (RMSEA) was calculated for each pair of correlations, along with the percentage of occasions in which RMSEA was lower than 0.1, the adequate value for parameter estimation (^{Hooper et al., 2008}).

To assess the factor structure of the scale, LISREL 12 was used to estimate the polychoric correlations and the asymptotic variance-covariance matrix. Standardized factor loadings were then calculated (where lambdas of .3 or higher were considered adequate) along with fit indexes (^{Holgado-Tello et al., 2019}; ^{Sanduvete-Chaves et al., 2018}), i.e., χ² (non-significant values -p > .05- allows the null hypothesis of model fit to be accepted, though large sample sizes tend to bias this index towards significant values). Other fit indexes calculated included the Root Mean Square Residual (RMR), where values below .05 were considered adequate); the Root Mean Square Error of Approximation (RMSEA), where values of .08 or lower were considered adequate; the Expected Cross Validation Index (ECVI), where values closer to the saturated model as opposed to the independent model were considered adequate; Critical N (CN), where values higher than 200 were considered adequate; the Parsimony Goodness-of-Fit Index (PGFI), where values between .5 - .9 were considered adequate. Finally, in the following indexes, values higher than .9 were considered adequate: the Normed Fit Index (NFI), the Comparative Fit Index (CFI), the Non-Normed Fit Index (NNFI), the Incremental Fit Index (IFI), the Relative Fit Index (RFI), the Goodness-of-Fit Index (GFI), and the Adjusted Goodness-of-Fit Index (AGFI).

Using JASP version 0.16, the reliability of each factor obtained was studied by calculating McDonald’s Omega (ω). Results higher than .80 were considered strong reliability evidence and .65 - .80, acceptable (^{Kalkbrenner, 2023}). For item discrimination, corrected item-total correlation coefficients were computed. Results were interpreted as excellent when values were higher than .40, good for values between .30 - .40, adequate for .20 - .30, and inadequate for < .20 (^{Barbero-García, 1993}).

Selection of Studies

Figure 1 summarizes the selection process of the papers for this study. The final sample of primary papers included in the study was 650.

Figure 1. PRISMA 2020 Flow Diagram for Systematic Reviews (^{Page et al., 2021}). 

Inter and Intracoder Reliability and Descriptive Statistics of the Items

Table 1 presents both inter and intracoder reliability. ICC coefficients were adequate, ranging from .73 to 1.

Table 1 also presents descriptive statistics. The median was .5 for most of the items, with 0 applying only to item 8 (Software). The means were between .13 and .98, the standard deviations ranged between .11 and .46, and there was no normal distribution of the items.

For items 11 (Data analysis) and 13 (Theoretical framework), means were over .9, which implies a low discrimination capacity. Items 11 (Data analysis), 12 (Objectives), 13 (Theoretical framework) and 14 (Units of study) showed standard deviation of 0.25 or below, which implies a low variability. Items 11 (Data analysis) and 13 (Theoretical framework) also showed an extreme negative skewness (over 3 points). Finally, item 11 (Data analysis) showed an extreme positive kurtosis.

To analyze the relationship between items, polychoric correlations were calculated. Table 2 presents the bivariate polychoric correlation matrix.

Items 12 (Objectives), 13 (Theoretical framework), 14 (Units of study), 15 (Sessions) and 16 (Discussion section) stood out from the other items since their correlations were negative and/or low (r = -.29 - .23). The highest positive bivariate correlations were between items 2 (Observation unit criteria), 3 (Temporal criteria), and 4 (Dimensionality criteria) (r = .98 - 1), but this triad also correlated with item 1 (Direct/indirect observation) (r = .82 - .84).

Study of Dimensionality

To obtain empirical evidence about the number of factors that the scale presented, a parallel analysis was done. Table 3 presents the result of this analysis.

Both the percentage of variance and the value of McDonald’s Omega (ω = .87) suggest only one factor to be considered. Nevertheless, the UniCo = .41 and the ECV = .31 lead data to be treated as essentially multidimensional (^{Ferrando & Lorenzo-Seva, 2018}).

Exploratory Factor Analysis

To study the factor structure of the scale, a principal-axis factor analysis with oblimin rotation was conducted on the 16 items. Bartlett’s test of sphericity and the KMO measure verified the sampling adequacy for the analysis, Χ ² = 0.839, p < .001, KMO = .84, and all the KMO values for individual items were above the acceptable limit of .5 (^{Field, 2018}; ^{Kaiser & Rice, 1974}). According to the initial analysis of eigenvalues for each factor in the data, five factors had eigenvalues over 1 and, in combination, these explained 62.23% of the variance. Based on the point of maximum curvature obtained in the scree plot (see Figure 2), two factors that jointly explained 40.81% of the variance were retained.

Figure 2. Scree Plot Resultant of the Exploratory Factor Analysis. 

Table 4 presents the rotated factor loadings matrix. As expected, based on the examination of the polychoric correlation matrix (Table 2), items 1 (Direct/Indirect observation), 2 (Observation unit criteria), 3 (Temporal criteria), and 4 (Dimensional criteria) presented high correlations and showed high loading values in the same factor. Items 6 (Data type), 7 (Observation instrument), and 9 (Type of parameters) showed similar loadings in both factors. On the other hand, cut-off point for factor loading was set at .3. Items 12 (Study objective), 13 (Theoretical framework), 14 (Observation units), 15 (Sessions), and 16 (Discussion), all of which had negative and/or low correlations with all other items, presented factor loadings below .3 for both factors and were excluded from the study.

The items that cluster on factor 1 were items 1 (direct/indirect observation), 2 (unit criterion of the observational design), 3 (temporal criterion of the observational design), 4 (dimensional criterion of the observational design), 5 (codification manual), and 6 (data type), suggesting that factor 1 represents the quality of the study design. The items that cluster on factor 2 were items 7 (observational instrument), 8 (software), 9 (type of parameter), 10 (data quality control), and 11 (data analysis), suggesting that factor 2 represents the quality of the measurement and analysis.

Confirmatory Factor Analysis

Based on the results of the parallel analysis, it was determined that a single factor should be retained, and that data should be treated as essentially multidimensional. This, in addition to the similar factor loadings that several items displayed in both factors, led to a decision to carry out a second-order confirmatory factor analysis.

Model Fit

The χ² test was significant χ² (42) = 531.79, p < .001, probably due to the sensitivity of this test to high sample sizes. RMR was 0.084, yielding a result slightly higher than expected. The other indexes showed an appropriate fit on the second-order factor model, RMSEA = 0.000, 90% CI [0.000, 0.000]; ECVI = 0.28 (saturated ECVI = 0.41; independent ECVI = 16.37); CN = 37964663941.58; PGFI = .62; NFI = 1; CFI = 1; NNFI = 1.01; IFI = 1.01; RFI = 1; GFI = .98; AGFI = .97.

Therefore, it seems logical to accept a factor structure comprised of a quality design factor and a quality measurement and analysis factor, both grouped under a second-order global methodological quality factor. Figure 3 shows the MQSOM structure with the standardized factor loadings.

Figure 3. Structure of the Methodological Quality Scale for Observational Methodology Studies with the Standardized Factor Loadings. 

Table 5 shows the descriptive statistics for each factor. Reliability yielded strong-acceptable evidence, and discrimination was excellent.

Table 5. Descriptive Statistics of the Factors. 

	First Order Factors		Second Order Factor
Descriptive statistic	Factor 1 Design quality	Factor 2 Measurement and analysis quality	General Factor (Methodological quality)
Mean	0.51	0.76	0.62
Standard deviation	0.32	0.21	0.24
McDonald's Omega	.9	.68	.87
Average discrimination	.46	.67	.55

^Note.Mean and standard deviation values range from 0 to 1.