Measuring SRL: The MSLQ

A major challenge in advancing research into SRL emerges from the measurement of its various components and self-regulatory processes (^{Curione & Huertas, 2016}; ^{Rovers et al., 2019}). Although other tools exist for the same purpose, the most commonly used questionnaire for SRL assessment is the MSLQ (^{Pintrich et al., 1991}) and it consists of two main dimensions: one focuses on motivation and the other one on learning strategies. Indeed, several Latin American studies have adapted the MSLQ into the Spanish language (^{Donolo et al., 2008}; ^{Inzunza et al., 2018}; ^{Muñoz, 2012}; ^{Ramírez et al., 2013}; ^{Ramírez-Echeverry et al., 2016}), but various formulations, derived from the Spanish language’s cultural nuances, make these adaptations differ between territories and generate difficulties in understanding items’ content.

The only adaptation for Spain is the “Cuestionario de Estrategias de Aprendizaje y Motivación II”, created and validated by ^{Roces et al. (1995)} and later by ^{Martínez and Galán (2000)}. Although CEAM II is considered a highly valuable tool (^{Credé & Phillips, 2011}), it is not exempt from some psychometric problems, the most noteworthy being: 1) ambiguity in some of its items’ formulation; 2) possible lack of discrimination between some dimensions, for example, time and study environment and effort regulation or peer learning and help seeking; 3) differences in reliability indices in some sub-dimensions of the original version; and 4) lack of consensus on the instrument’s internal structure or dimensionality between studies.

Gender differences in the MSLQ

Together with the issues derived from the assessment of SRL, increasingly more attention is paid to factors that differentiate successful students from the unsuccessful ones, among the more and more diversified student body in modern educational contexts (^{Li, 2019}). In fact, one of the individual factors that was already urged to be investigated by ^{Pintrich and de Groot (1990)}, and increased attention over the years is related to gender (^{Torrano & Soria, 2017}). Despite studies that found no significant gender differences (^{Bruso & Stefaniak, 2016}; ^{Syam et al., 2016}) and even some which postulated that men had higher scores in critical thinking (^{Rodarte-Luna & Sherry, 2008}) and deep processing, most found differences favoring women, both in motivational components and learning strategies (^{Bozpolat, 2016}; ^{Torrano et al., 2017}; Torrano & Soria, 2017; ^{Tseng et al., 2017}). Specifically for motivation, studies found that women had higher intrinsic goal orientation, more test anxiety (^{Albert, 2017}), more beliefs of self-efficacy (^{Rianudo et al., 2006}) and control (^{Navea, 2015}), and lower levels of extrinsic goal orientation (^{Rusillo & Casanova, 2004}). As for learning strategies, women scored higher in planning, goal setting, organization (^{Valenzuela & Suarez, 2017}), metacognitive self-regulation (Albert, 2017; Suarez et al., 2004), personal regulation, and control of the context (Navea, 2015; ^{Velasco & Cardeñoso, 2020}; ^{Zimmerman & Martinez-Pons, 1990}).

In sum, given the MSLQ’s great usefulness and its widespread use in national and international educational contexts, we recognize the need to adapt it to the current variable and uncertain circumstances along with the analysis of gender differences so as to contribute to the understanding of self-regulated learning. Therefore, the aims of this study are: (1) to adapt the MSLQ into Spanish for Spain and validate its scoring for its use with university students; and (2) to assess the appropriateness of the Spanish MSLQ’s use for both men and women in order to analyze the possible differences between genders in the SRL.

Participants

Through incidental sampling, researchers recruited 456 Spanish university students who completed the MSLQ, only 428 answered all the items, 28 left some unanswered, with 24 different and random patterns: 18 had only 1 blank item, 4 had 2, 1 had 3, and 1 had 4. Questionnaires with missing values were omitted from subsequent analyses. 314 (73.7%) of the students enrolled in the study were women and 112 (26.3%) were men, whose socio-demographic characteristics can be seen in the following table (Table 1).

Table 1 Characteristics of participants 

Differences between men and women did not achieve statistical relevance in Pearson chi-square tests and were not associated with moderate to high values in Cramer's V-test.

Adaptation process

Following ^{International Test Commission (ITC, 2017)} guidelines, the questionnaire’s intellectual property rights were checked, and the process of linguistic, conceptual, and cultural adaptation was conducted. To this end, a multidisciplinary team of four bilingual English-Spanish educational experts was formed. First, two team members translated the original English version into Spanish. Next, the other two, blind to the original English version, back-translated the first Spanish version into English. The team evaluated similarities and discrepancies, considering Hambleton and Zenisky’s list for quality control of the items’ translation-adaptation (²⁰¹¹). To assess the adapted questionnaire’s understandability, legibility, and duration, researchers modeled the instrument and conducted a pilot test with university master’s degree students in psychodidactics.

Collection of information

After obtaining informed consent from all participants and according to current regulations, researchers administered the test.

Data analysis

The initial analysis evaluated the presence and patterns of missing and atypical values and whether basic assumptions underlying the general linear model were met.

Subsequently, to study the relations between MSLQ items and their concordance with the theoretical model of its construction, descriptive statistics were calculated for each item (e.g., % cases chosen by each option, mean standard deviation, asymmetry, kurtosis, and corrected homogeneity indices), and confirmatory factor analyses (CFAs) were conducted to evaluate the relationship pattern between items and their sub-dimensions. In addition, a formal description of the resulting sub-dimensions was developed, including average variance extracted (AVE) and composite reliability (CR). Given that the scores’ distribution distances with respect to the normal curve were small, the estimation method used in CFAs was of maximum likelihood. Evaluation of the model fit to data was based on the χ² test, χ²/df ratio’s value, together with information provided by the incremental goodness-of-fit index (CFI), the root mean square error of approximation (RMSEA), and its standardization (SRMS). Acceptable models met the following criteria: χ² /df = >5; CFI=0.90+; and RMSEA and SRMS=0.08 or less (^{Hu & Bentler, 1999}; ^{Kenny et al., 2015}).

Several models were then piloted to test relationships between dimensions derived from previous analyses.

Next, once the model with the best fit was selected, possible gender differences were analyzed, specifically, an analysis of progressive invariance of associations between men and women in the MSLQ’s components was carried out. Equivalence levels were defined according to parameters conditioned as equal in the study’s groups. The simplest model was configural invariance (factorial loads pattern); which was evaluated by adding constraints, metric invariance (factorial loads magnitude), scalar (intercepts or covariances), and strict (residual variances). To accept configural, metric, scalar and strict invariance, a triple criterion was used: differences in chi-square values (should be non-significant), Akaike's information criterion (AIC) (the smaller - the better), and in CFI (must be equal to or less than 0.01) between two immediate models.

Finally, comparisons of mean differences between male and female participants were developed using the student’s T test. Associated effect sizes were calculated by Hedges’ g (g’), with reference values 0.20, 0.5, and 0.8 as small, medium, and high effect sizes; respectively.

Analyses were conducted using SPSS and AMOS, version 24.0.

Preliminary analysis, evaluation of measurement models, dimensions and components’ metric properties in the MSLQ’s Spanish version

First, the properties of each of the items that make up the test were analyzed to know the number of missing, outliers, the distribution of scores and individual psychometric indices. Then, factor analyses were conducted to evaluate each sub-dimension’s unidimensionality. Results led to 19 items’ suppression due to their low relationship with their underlying factors (3 motivational (9.7% of component items); 16 learning strategies (32%)).

The Table in appendix 1 summarizes the formal description of the final, validated Spanish MSLQ, consisting of 66 items. Few items had a high floor or ceiling effect, but that would be expected in those that did, due to their content. On all items, averages were slightly above the theoretical average of four points; in statistics of asymmetry and kurtosis, they were from −1 to 1 on most items, thus reporting a score distribution similar to a normal curve. Factor weights were greater than .40 in all cases and .50 in most cases. Corrected homogeneity indices were good, higher than 040 in most cases. These results endorsed the individual suitability of each element on the questionnaire.

Second, the joint measurement model was tested for all sub-dimensions in each component, resulting in the creation of two additional components, extraction of extrinsic objective and anxiety dimensions of the motivation component, along with restructuring of the sub-dimensions in the learning strategies component.

For the motivation component, four sub-dimensions were extracted. The first, called intrinsic goal orientation, was composed of items that formed that sub-dimension in the original version (1, 16, 22, and 24). The same occurred in the second sub-dimension, task value, formed by items 4, 10, 17, 23, 26, and 27. The third sub-dimension, control beliefs, was formed only by items 2 and 18. The fourth, self-efficacy for learning and performance, replicated the sub-dimension’s original version (items 5, 6, 12, 15, 20, 21, 29, and 31).

In the learning component, a solution of five sub-dimensions was obtained. First, organization of study material was constituted by items belonging to organization (32, 42, 49, and 63) and rehearsal scales (46, 59, and 72) and focused on the material’s self-organizing aspects. The second, deep learning, contained questions about relating, developing, questioning, or establishing connections between ideas, concepts, or conclusions (items 53, 62, 64, 69, 81, 38, 47, 51, 66, and 71) and combined the original scales of elaboration and critical thinking. The third sub-dimension, metacognitive self-regulation, matched the original and was formed by items 36, 41, 44, 54, 55, 56, 76, 78, and 79. The fourth, time and effort management, was composed of items from the original subscales of time and study environment (43, 52, 70, and 77) and effort regulation (37, 48, 60, and 74). The sub-dimension in relation to peers was composed of items 34, 45, and 50 of the peer learning scale and of item 68 of the help seeking scale.

Items belonging to the sub-dimensions of anxiety and extrinsic goal orientation, originally in the motivation component, were grouped into 2 one-dimensional components. Thus, the extrinsic goal orientation component overlaps the original version (items 7, 11, 13, and 30). Similarly, items in the anxiety component were also associated in an independent one-dimensional factor (3, 14, 19, and 28) in accordance with the original version-except for item 8, which was eliminated.

For each of these four measurement models, Table 2 displays adjustment indices that guarantee their adequacy to the data.

Table 2 Component Adjustment Indexes of the MSLQ (n = 428) 

Table 3 displays descriptive statistics for the nine sub-dimensions and the four components. It also includes information on their scores’ reliability. In brief, results showed that all components and sub-dimensions were distributed similarly to the normal curve, presenting only slight negative asymmetry. For reliability, both Cronbach’s alpha and CR showed moderate and high internal consistency levels. Some AVE indices were lower than desirable.

Table 3 Descriptive Statistics and Internal Consistency of Components and Sub-dimensions 

As for relationships between components, model fit, in which all components showed correlation, obtained satisfactory test results. However, modification indices revealed that adjustment increases significantly if we allow the motivation component to explain deep learning. The final model is the following, due to its good theoretical sense (Figure 1): (ꭓ² = 194.2; p < .001; ꭓ²/df = 3.18; CFI = .92; RMSEA (IC90%) = .09 (.08-.12), SRMR = .05).

Figure 1 Model of relations between the components of the Spanish version of the MSLQ in men and women 

Regarding subgroup analysis, progressive estimation of invariance began with the configural invariance model. The adjustment indexes obtained (Table 4) allowed to accept the equivalence of the model between genders. Adding restrictions on the regressing coefficients, the values that are listed in the table, and the differences between ꭓ² (Δꭓ ² =11.33; p = .183), AIC (ΔAIC = 4.66), CFI (ΔCFI = .001) and RMSEA (ΔRMSEA = -.003) led us to accept the metric invariance model, which allows us to assess the equivalence between interceptal values. The values obtained permits us to reject this model, both by independently evaluating it and by analyzing it with respect to its nesting with the metric invariance model (Δꭓ² = -93.88; p < .001; ΔAIC = -76,48; ΔCFI = -.050, ΔRMSEA = -.011). Comparing the estimated intercepts for both groups, an attempt was made to achieve partial scalar invariance by freeing up parameter constraints for subdimensions that showed more differences. Since it was unsuccessful, we decided to stop the analysis.

Table 4 Adjustment indices for factor invariance of the Spanish MSLQ model by gender 

In summary, factor invariance analyses indicated feasible comparisons between men and women, given that the minimum requirement of metric invariance in the tool’s gender structure and loads was met.

Gender differences

In relation to gender comparisons, as can be seen in Table 5, out of a total 13, 10 comparisons showed statistically significant differences between men and women. In the motivation component, however, the associated effect size was moderate only in terms of task value, whereas in learning strategies, the effect size was moderate or high in all sub-dimensions except deep learning. The greatest differences occurred in organization of study material and management of time and resources. Similarly, for test anxiety, women presented significantly higher scores than men, with moderate effect size.

Table 5 Gender differences 

Adaptation and psychometric properties of the MSLQ

First, the systematic and rigorous translation procedure led to a MSLQ version adapted to the sociolinguistic reality of the Spanish territory. This version is semantically equivalent to the original version, thus overcoming possible comprehension problems in the use of Latin American versions. Specifically, as we described before, in some cases certain terms have been changed. Also, modifications have been made with regard to specific expressions or grammatical aspects, such as verb tenses or the order of the sentence

Analysis of the Spanish version’s internal structure allowed us to conclude: 1) the final MSLQ Spanish version is shorter (66 items) than the original (81 items); 2) considered individually, each of the 66 items has adequate metric properties; and 3) the four-dimensional structure obtained better fit indices than alternative models (i.e., original structure, one-dimensional model, two-dimensional, five-dimensional oblique and orthogonal and nine-dimensional oblique and orthogonal for each component) even though it does not correspond to the original instrument’s structure.

The new Spanish version is structured of four components, two multidimensional and mutually related: motivation (intrinsic goal orientation, value given to the task, control beliefs, perceived self-efficacy) and learning strategies (organization of study material, deep learning, metacognitive self-regulation, time and effort management, help seeking and relationships with peers). The other two structures are one-dimensional, mutually related, and independent of the others-test anxiety and extrinsic goal orientation.

Specifically for motivation, four sub-dimensions were extracted. The first, intrinsic goal orientation, was composed of items forming the original version’s subscale (^{Pintrich et al., 1991}); this is in partial agreement with ^{Roces et al. (1995)} because item 24 was part of the subscale task value, but in opposition to what ^{Cardozo (2008)} obtained, since this subscale had no factor. In the adaptation by ^{Martínez and Galán (2000)}, the factor’s items were dissolved into subscales of self-efficacy for learning and performance and task value.

Similarly, the second sub-dimension, task value, also corresponded to the original version. Other adaptations found that this factor’s items were grouped with others belonging to the scales of test anxiety, intrinsic goal orientation, and control beliefs (^{Martínez & Galán, 2000}) or self-efficacy for learning and performance (^{Cardozo, 2008}).

The third sub-dimension, control beliefs, was formed by two of the four items belonging to that subscale in the original version. ^{Martínez and Galán (2000)} also found this factor to be composed of two items, while ^{Roces et al. (1995)} reproduced the original subscale.

The fourth sub-dimension, self-efficacy for learning and performance, replicated the original version’s subscale, coinciding with ^{Roces et al. (1995)} but differing from ^{Inzunza et al. (2018)} because it was fragmented into two subscales.

Items belonging to sub-dimensions of test anxiety and extrinsic goal orientation were grouped into two independent components, although they were congruent with the original. However, this excludes item 8 in anxiety, which was eliminated in accordance with ^{Inzunza et al. (2018)}.

Regarding learning strategies, a solution of five sub-dimensions was obtained. Some were congruent with the structure of ^{Pintrich et al. (1991)}, but others were clustered around common topics that did not follow this structure, such as in CEAM II’s adaptation to diverse contexts (^{Cardozo, 2008}; ^{Martínez & Galán, 2000}; ^{Roces et al., 1995}) and in other Latin American adaptations (^{Ramírez-Echeverry et al., 2016}). The first sub-dimension was organization of study material, items of which belonged to scales of organization and rehearsal in ^{Pintrich et al.’s model (1991)}, thus coinciding with previous studies’ findings (^{Martínez & Galán, 2000}; ^{Roces et al., 1995}).

The second sub-dimension, deep learning, included items about relating, developing, questioning, or establishing connections between ideas, concepts, or conclusions; it contained the original scales of elaboration and critical thinking. This sub-dimension coincided, to a large extent, with CEAM II’s elaboration scale (^{Roces et al., 1995}) and with other studies’ findings (^{Cardozo, 2008}; ^{Ramírez-Echeverry et al., 2016}).

The third sub-dimension, metacognitive self-regulation, referred to the degree of awareness, knowledge, and control of cognitive aspects when planning, monitoring, and regulating study; it used ^{Pintrich et al.’s (1991)} original items. Our adaptation included two more items (78, 79) than ^{Roces et al.’s (1995)} and had a more solid structure than ^{Martínez and Galán’s (2000)}.

The fourth sub-dimension, time and effort management, focused on contextual and behavioral aspects that represent an obstacle or difficulty in achieving academic goals. Its items belong to the original subscales of time and study environment. This sub-dimension completes ^{Roces et al.’s (1995)} adaptation since it adds more items.

The last sub-dimension was relationship with peers which concerns learning with peers and resorting to them if needed. Thus, its items belong to peer learning and help seeking, coinciding with eds. Thus, its items belong to peer learning and help seeking, coinciding with previous research (^{Cardozo, 2008}; ^{Inzunza et al., 2018}; ^{Ramírez-Echeverry et al., 2016}; ^{Roces et al., 1995}).

In line with previous studies, some items were eliminated due to inadequate psychometric properties. Specifically, on the motivation scale, items 8, 9, and 25 in the test anxiety and control beliefs scales, respectively, were deleted. On the learning strategies scale, several items assessing behavioral and contextual regulation were eliminated. They belonged to the subscales of time and study environment (35, 65, 73, and 80) and help seeking (40, 58, 75), in accordance with ^{Roces et al. (1995)} and ^{Ramírez-Echeverry et al. (2016)}. Likewise, items related to metacognitive self-regulation (33, 57, 61), rehearsal (39) and elaboration (67) were removed.

In spite of these dimensions’ restructuring and certain items’ elimination, we believe that our new structure does not affect the base model’s theoretical coherence. Similar to our findings, ^{Credé and Phillips (2011)} suggested a four-component solution in their university meta-analytic MSLQ study. Their first and second components were composed by learning strategies, the third by motivational aspects, and the fourth by test anxiety. For our version’s reliability, all components and sub-dimensions have adequate internal consistency, with values estimated for Cronbach’s alpha and for CR higher than those in other studies. However, indices of AVE were lower than desirable in some sub-dimensions.

Gender differences

This study’s second objective was to analyze the Spanish MSLQ’s adequacy between men and women in order to analyze the possible differences between genders in SRL. Thus, factor invariance was evaluated by adding restriction on loads in the factor, intercepts, and error variances so that they were equal between groups. Results supported metric invariance in the instrument’s structure according to gender; therefore, gender differences were analyzed.

As expected, results showed gender differences in 10 of the 13 comparisons: moderate and high associated effect size in sub-dimensions of intrinsic goal orientation, task value, test anxiety, material organization, deep learning, metacognitive self-regulation, time and effort management, and relationship with peers. These findings coincide with previous research (^{Albert, 2017}; ^{Navea, 2015}; ^{Suárez et al., 2004}; ^{Valenzuela & Suárez, 2017}). No statistically significant differences were found in control beliefs, self-efficacy for learning and performance, and extrinsic goal orientation. This endorses the need for more study of gender differences in self-regulation in learning, as ^{Torrano and Soria (2017)} stated.

Limitations, practical implications, and future lines of study

This study’s results are somewhat conditional. First, the incidental sample’s size was not very large, which led us to perform the AFE and AFC on the total data. In addition, all participants belonged to a faculty of education, so given the importance of university students’ motivation and learning strategies, the number of students, faculties, and disciplines should be increased. To understand more about SRL processes, this quantitative research could be enriched with qualitative study, thus adding specific information about global cultural idiosyncrasies among university learners.

With a view to future research, as pointed out in the revision of the MSLQ of ^{Curione and Huertas (2016)}, it is worth highlighting the need to adjust the questionnaire to the) suggested in their recent MSLQ revision, adjusting measurement instruments to align with contemporary social and technological changes that have occurred in recent years, adding sub-dimensions that includes an ongoing need. Future psychometric studies might add situations that students perceive as emotionally and academically significant, such as group work, organization of tasks, and anxiety before oral presentations (^{Larruzea-Urkixo et al., 2020}, ²⁰²¹).