SciELO - Scientific Electronic Library Online

 
vol.34 issue3Stability of Measure In High Intellectual Ability: preliminary resultsBrief Resilience Scale in front children's Behavior Problems (BRS-BP) author indexsubject indexarticles search
Home Pagealphabetic serial listing  

My SciELO

Services on Demand

Journal

Article

Indicators

Related links

  • On index processCited by Google
  • Have no similar articlesSimilars in SciELO
  • On index processSimilars in Google

Share


Anales de Psicología

On-line version ISSN 1695-2294Print version ISSN 0212-9728

Anal. Psicol. vol.34 n.3 Murcia Oct./Dec. 2018

https://dx.doi.org/10.6018/analesps.34.3.299041 

Developmental and Educational Psychology

The dimensional validation of the Student Engagement Questionnaire (SEQ) with a Spanish university population. Students' capabilities and the teaching-learning environment

Validación dimensional del Student Engagement Questionnaire (SEQ) en población universitaria española. Capacidades del alumno y entorno de enseñanza/aprendizaje

Bernardo Gargallo1  , Jesús M Suárez-Rodríguez2  , Gonzalo Almerich2  , Irene Verde1  , Mª Àngels Cebrià i Iranzo3 

1 Departamento de Teoría de la Educación. Universidad de Valencia (Spain)

2 Departamento de Métodos de Investigación y Diagnóstico en Educación. Universidad de Valencia (Spain)

3 Departamento de Fisioterapia. Universidad de Valencia (Spain)

Abstract:

In this work, the SEQ (Student Engagement Questionnaire - Kember & Leung, 2009) was validated with a Spanish sample. This instrument is designed to make a joint assessment of teaching-learning processes at university and to provide feedback to teachers and institutions to improve these processes. A sample of 805 subjects from three Valencian universities was used. The questionnaire assesses the capabilities of several students, and the teacher’s ability to design an adequate learning environment. Several versions have been developed until its dimensions (latent variables), and the relations between the involved variables, achieve high degree of stability. A confirmatory factor analysis corroborated the structure proposed by the authors and the relations between the involved variables. Hence the instrument seems suitable to be used in Spanish samples for the proposal it was designed for.

Keywords: Assessment of teaching-learning processes; Questionnaire; Students’ capabilities; Teaching and learning environment; University students

Resumen:

En este trabajo se validó el Study Engagement Questionnaire (SEQ) de Kember y Leung (2009) en una muestra española. Es un instrumento diseñado para realizar una evaluación conjunta de los procesos de enseñanza-aprendizaje en la universidad y devolver feed-back a los profesores y a las instituciones para la mejora de estos procesos. Se utilizó una muestra de 805 sujetos de tres universidades valencianas. El cuestionario evalúa diversas capacidades del alumno así como la capacidad del profesor para diseñar un entorno de aprendizaje idóneo para aprender. Ha tenido varias versiones hasta conseguir una estabilidad notable en sus dimensiones (variables latentes) y en las relaciones entre las variables que contempla. Mediante análisis factorial confirmatorio se ha corroborado la estructura propuesta por los autores así como las relaciones entre las variables involucradas. El instrumento parece, pues, adecuado para ser utilizado en muestras españolas para el propósito para el que fue diseñado.

Palabras clave: Evaluación de los procesos de enseñanza-aprendizaje; Cuestionario; Capacidades del alumno; Entorno de enseñanza y aprendizaje; Alumnos universitarios

Introduction

Conducting quality research into Higher Education requires good measuring instruments, which is particularly important in the area of teaching-learning processes. To suitably diagnose both processes and to undertake improvement actions, a suitable assessment needs to be made.

Despite them being two different processes, teaching and learning are absolutely interrelated (Sampascual, 2010) as teaching is done to promote students’ learning (Coll, 2008a; Mayer, 2010).

Both processes are complex as many variables, aspects or factors intervene in them, and it is practically impossible to identify, record and describe them all. Thus focusing on particularly relevant ones to explain and understand these processes is fundamental (Coll, 2000).

The learning process can be interpreted as an active, constructive and significant process (González-Pienda, 1999) through which knowledge, skills, conducts, attitudes, values, etc., are acquired as a result of studying, experience, teaching, reasoning and/or observation. Learning provides students with long-lasting changes in their knowledge, skills, attitudes, etc. (Mayer, 2010).

Although learning is an individual activity, it is done in a social-cultural context (Coll, 2000) and is the result of cognitive processes that allow new information to be assimilated and to internalise new information (in relation to facts, concepts, procedures, values, etc.) and form new significant and functional mental representations (Kintsch, Franzke & Kintsch, 1996) that allow learners to act in a new way; apply what has been learned to other situations (Vidal-Abarca, 2010). Learning does not simply consist in memorising information, but other complex cognitive operations are also necessary, such as understanding, applying, analysing, summarising, assessing, etc. (Monereo, (1990).

The interpretation of what learning means has changed with time. Mayer (2010) distinguish three times/three metaphors to explain the process, which came into play in the 20th century and still apply in the present-day: learning as in acquiring or intensifying responses (conductism in the first half of the 20th century), knowledge acquisition (cognitive psychology/information processing theory in the 1950s and 1960s), and knowledge building (constructivism and socio-cognitive approaches, as of the 1970s).

By acknowledging that the three metaphors explain relevant elements of learning, we feel more closely identified with the third one, which understands that apprentices actively form tailor-made mental representations, which confer their experiences sense. Thus the focal point lies in learners, where the teacher acts as a guide in the process, a mediator who helps students build knowledge and in an increasingly more autonomously way.

Understanding learning in this way implies that learners must improve their learning strategies and self-regulation mechanisms which they have to manage their own learning processes in such a way that autonomy and self-regulation are gained (Zimmerman, 2002 and 2008) in the lifelong learning context. This demands the development of competences/capabilities/skills,1 particularly in the university where students are adult learners, in a society in which permanent learning is needed. In fact the curricular design of university degrees devised according to the Bologne process, each with a different name, has included a relevant repertoire of generic/transversal competences of this kind (García Espejo & Ibáñez Pascual, 2006), which are useful for several learning or professional tasks, and allow them to be transferred to several areas (García, Díaz, Ramírez & Castro, 2008). Of these, we are particularly interested in those related with management and with looking at learning processes in-depth. Some examples of such skills are critical and self-criticism capability, autonomous/self-regulated learning, planning, organisation and managing time, efficient information mangement, expression/communication skills, interpersonal skills and teamwork, problem solving, creativity and innovation, etc. (García, Díaz, Ramírez & Castro, 2008; Tuning, 2009). This has been generalised concern in the countries that share the Bologne convergence process, which has also emerged in many other education systems, which attempts to specify generic skills with which graduates must leave university with to be able to face lifelong learning challenges (Barrie, 2006; Kember & Leung, 2009 and 2011; Longworth & Davies, 1996; Tait & Geoffrey, 1999).

Like the learning process, the teaching process is multidimensional and involves actions being taken by the teacher or the person who teaches by means of which this person shows or presents educational contents (knowledge, habits, rules, abilities, techniques, attitudes, etc.) to a student or group of students via certain methods and resources in line with the given objectives, and within a context so that students incorporate them into their mind and conduct (Mayer, 2010).

From a traditional viewpoint, this process can be interpreted as a process by which these data, habits, abilities, rules, techniques etc., can be transmitted by the person who knows (i.e, the teacher), where students are understood to act as receivers of such contents (Doménech Betoret, 2007).

From a more advanced and better adapted perspective, just as the constructivism and the sociocognitive perspective do, it can also be understood as a help, support, guidance and orientation process by which the teacher designs enriched settings for learning (Mayer, 2010), and offers learning opportunities, various means and resources so that students well prepare, organise, internalise and significantly incorporate information into their cognitive structures with the teacher’s help and mediation. This helps build knowledge and students are interpreted as active and constructive subjects (Coll, 2008b). What all this means is that the teacher is expected to provide an ample repertoire of skills and abilities with which one is able to manage the process.

As with the learning process, the interpretation of what the teaching process is and what it implies have changed with time according to the evolution that results from the devised learning theories (the teaching and/or instruction theory is based on, or must be based on, the learning theory) (Ertmer & Newby, 2013), and from the results obtained in research into the teaching processes, progressively demanding a teacher to be good, mastering a series of teaching skills, capabilities and competences, along with certain personal characteristics. The scientific production that stems from all this abounds at the university (Barrie, 2006; Cajide, 1994; García Ramos, 1997a and b, 1998; Gargallo, Sánchez Peris, Ros & Ferreras, 2010; Ibáñez-Martín, 1990 and 2001; Knight, 2005; Marsh, 1987 and 2007; Monereo & Domínguez, 2014; Monereo & Pozo, 2003; Ramsden, 1991; Rodríguez Espinar, 1993; Tait and Goffrey, 1999; Tejedor, 1993; Torra et al., 2013; Villar Angulo & Alegre de la Rosa, 2004; Zabalza, 2003 and 2007), which has allowed to shape an efficient repertoire of teaching skills, capabilities and/or competences of an efficient teacher, generally located in a series of main areas: planning teaching, managing teaching-learning process activities by several teaching methods that adapt to the objectives and the context, assessment abilities, communication abilities, the interpersonal relationship, tutoring, controlling and managing the classroom atmosphere, the innovation skill, teamwork skills, handling ICT, etc.

Given the complexity of both processes, as we mentioned at the beginning it is virtually impossible to assess both in an all-round complete way by contemplating all their integrating dimensions and elements. Thus in scientific dealings, resorting to sufficiently rich constructs that provide relevant information about them has been the more usual practice.

Therefore in the relationship with the learning process, explanatory constructs of its relevant elements have been devised; e.g., study skills and techniques, learning strategies, learning approaches, learning styles, and students’ capabilities, competences, academic performance, etc.

Something similar happens in the relationship with the teaching process, in which teaching styles, teaching models/guidelines, skills, capabilities and teaching competences, etc., have been specified.

We wish to point out that when we talk about assessing learning processes, we do not refer to the assessment that teachers make of the process and the resulting learning outcome of the teacher’s students which ends up in a qualitative and/or a quantitative assessment. Instead we refer to examining in-depth the processes that students mobilise to learn through the constructs that make them operative which, in most cases, are also the result of the learning process. This takes place, for example, with skills, learning strategies, and the capabilities or skills which, in the end, are both the independent and dependent variables of this learning process.

When it comes to assessing teaching/learning processes, and analysing their relationship, several possibilities in the research area come into play. Of all these possibilities, one possible option is plausible and is included in the literature: assessing the relevant constructs of students’ learning on the one hand (learning strategies, approaches, styles, skills, etc.) and the teaching processes implemented by the teachers who teach them on the other hand (methods, styles, teaching capabilities/skills, etc.) by using students and teachers as sources of information, and then crossing the data from both measures in an attempt to explain the influence of the teaching and assessment process on the learning process. This has been the option that several studies have taken: Gow and Kember (1993), Kember and Gow (1994), Trigwell and Prosser (2004), Trillo and Méndez (1999), and Gargallo and team (Gargallo, 2008; Gargallo, Garfella, Pérez & Fernández, 2010).

To be able to proceed in this way, a vast repertoire of assessment instruments of both teaching-learning processes, which quite often come in a self-reporting style, is available. It is neither our intention to include a list of them herein, nor to make a critical analysis of them because they are available in the literature.

In this work however, we are interested in dealing with the joint assessment of the learning-teaching processes in the university setting with a single instrument and using students as the source of information to specify to what extent the learning-teaching environment designed by teachers influences students’ acquiring generic capabilities/competences.

We are interested in an instrument that collects information about students’ development of generic/transversal capabilities/competences by taking into account the relevance of the construct and the growing importance that acquiring them has in today’s education systems for university graduates, and by adapting them to the current university context coherently with both Bologne process assumptions and current university syllabi. We are also interested in collecting information about the teacher’s capabilities/competences in relation to setting up a good teaching/learning environment for the same reason. As we previously mentioned, in both cases we have a large repertoire of capabilities/skills that also coincide considerably in different education systems in several countries.

Using students as a source of information is very interesting from our viewpoint because students’ perception of the teaching process managed by their teachers does not always coincide with that which the teachers have. Moreover, this is an “economic” consideration as it allows us to collect information about both processes from a single group of involved parties, students, which simplifies researchers’ work. Evidently, crossing the information provided by students with that provided by teachers would be ideal, and there are other instruments and procedures available for this purpose.

An example of such instruments, which work with students who act as a source of information to assess the teaching process by their teachers, is the Course Experience Questionnaire (CEQ) (Ramsden, 1991). The CEQ is widely used in the literature which, in its first version, assessed students’ perception of effectiveness of teaching using five scales that assess teaching skills: good teaching, clear goals, adequate workload, suitable assessment and emphasising independence. Other versions exist (Wilson, Lizzio & Ramsden, 1997), which are usually differentiated by adding the number of items to the questionnaire name (CEQ36, CEQ30, CEQ23). To the first five scales, another with generic skills is then added, which refers to the skills that a course helps students to develop or reinforce, and occupies six items in the 36-item version.

Despite being a good instrument, it has some limitations that result from considering only five or six factors. There is only one scale for generic skills which refers to those that students develop: problem solving, teamwork, planning, etc., but with only six items, and only five other factors to assess teaching-learning processes/teaching skills. From our point of view, the assessment instruments for teaching skills should include more scales given the complexity of the process and the many dimensions that teaching entails (Feldman, 1996; Kember, Leung & Ma, 2007; Marsh, 1987, 2007; Mas, 2012; Zabalza, 2003). This instrument either does not collect students’ perceptions of their own learning process and of the influence that the teaching process has on them, which happens in the first versions, or clearly assesses this learning process insufficiently, as previously mentioned, only with a 6-item scale in later versions.

In this context, and in order to overcome the above instrument’s limitations, Kember and Leung (2005a; 2005b; 2009) developed the Student Engagement Questionnaire (SEQ).

Student Engagement Questionnaire (SEQ): devising and validating it

The purposes of Kember et al when they devised this instrument were to: make a sufficiently rigorous diagnosis to identify the strong and weak points in the teaching-learning process; return feedback to both teachers and institutions to improve it; for it to be consistent with the research conducted on the teaching-learning environment; to collect the necessary constructs to assess it.

The questionnaire had to cover two main dimensions/theme blocks:

  • - Analyse the capabilities that university students have to acquire during the process

  • - Assess the learning environment that the teacher creates in class to make it easy to acquire

In order to design the repertoire of capabilities, the author employed a work which used panels of teachers from different faculties who made lists of the attributes that the graduates of their faculty required. The results found by this procedure were compared with other works in the scientific literature, such as those reviewed by Pascarella and Terenzini (1991), and both indicated a high degree of communality. From this point, banks of items were devised to assess the selected capabilities, which were adjusted, refined and reduced in successive validation iterations. This was to be the first part of the questionnaire.

The second part should cover the characteristic elements of the teaching-learning environment that the teacher sets up to achieve good learning. In principle, the constructs included in this second part of the questionnaire were taken from the literature available on students’ experience with the teaching-learning process. The first versions of the instrument included only a few scales as it was considered that teachers’ teaching had been sufficiently assessed. However in following versions, the scales in this second part increased in number.

During subsequent validation processes, the different versions included a variable number of dimensions/scales/variables/plots included in the two main blocks mentioned earlier (students’ capabilities and the teaching-learning environment) which were maintained in them all (Kember & Leung, 2005a; Kember & Leung 2005b; Kember, Leung & Ma, 2007) (Table 1).

The last two works allowed a stable structure to emerge, which was maintained in later works (Kember & Leung, 2009 and Kember & Leung, 2011) (Table 1). This structure includes five latent variables/dimensions: two in the capabilities area; Intellectual Capabilities and Working Together; and three in the teaching-learning environment; Teaching, the Teacher-Student Relationship and the Student-Student Relationship). Likewise, it includes 15 observed variables/plots/scales included in the above-mentioned dimensions, specifically seven students variables and eight environment variables.

Table 1: The SEQ questionnaire dimensions and variables. 

These authors’ proposal of the variables is coherent with the previously available theory and, in relation to students’ capabilities and to the variables that the questionnaire assesses, with various proposals about students generic capabilities/skills that we mentioned before. They also have to do with autonomous and self-regulated management of teaching processes, and with communication skills, the interpersonal relationship, problem solving, etc. It is true that the questionnaire does not include all those involved/developed during the learning process, but does include a good sample of them. The same can be stated of the teaching-learning environment, which includes a reference about using teachers’ relevant capabilities/skills, which we also mentioned earlier, by contemplating the basic elements of handling the process: teaching for understanding and reinforcing active learning, quality assessment, use of feedback, communication skills and interaction, using cooperative learning, etc.

The model hypothesised and validated by the authors (Kember & Leung, 2005b; Kember, Leung & Ma, 2007; Kember & Leung, 2009) presents the theoretical structure of dimensions and the relationships among them, which are represented in Figure 1. This figure includes the coefficients that these authors found (Kember & Leung, 2009).

With this model, Kember and Leung (2009) attempt to analyse how the teaching-learning environment influences the development of students’ capabilities. The type of evidence from the construct that the authors provide is that of nomological validity (Hair, Black, Babib & Anderson, 2010; Kember & Leung, 2009).

For this purpose, the eight teaching-learning environment scales act as latent indicators of the three aforementioned latent constructs/dimensions (Teaching, the Teacher-Students Relationship and the Student-Student Relationship). According to these authors, these three constructs in the proposed model also correlate with one another (Kember & Leung, 2005b; Kember, Leung & Ma, 2007; Kember & Leung, 2009).

Regarding the seven students’ capabilities that make up the two dimensions (Intellectual Capabilities and Working Together) of the student variables, a relationship of the influence of Working Together with Intellectual Capabilities is established (Kember & Leung, 2005b; Kember et al., 2007; Kember & Leung, 2009).

The proposed model also presents an explanatory link of the latent dimensions from the teaching-learning environment with the constructs of students’ capabilities. Indeed a connection is formed between the influence of Teaching with the two latent variables of Students’ Capabilities (Kember & Leung, 2005b; Kember et al., 2007). Likewise, a link of the influence of the Student-Student Relationship is established with Working Together (Kember & Leung, 2005b; Kember et al., 2007; Kember & Leung, 2009), but the connection linking the Student-Student Relationship and Intellectual Capabilities is an indirect one via Working Together (Kember & Leung, 2005b). Kember and Leung (2005b) found no direct link in this case.

Nor did these authors find a direct link between the Teacher-Students Relationship and the two latent variables of Students’ Capabilites (Kember & Leung, 2005b). This relationship has an indirect effect on these capabilities (Kember & Leung, 2005b), and Kember and Leung (2005b) point out that it interrelates with the teaching-learning variables as a prerequisite prior to teaching.

NB: all the model’s parameters are significant; p ≤ 0.05

Figure 1: Hypothesised model. 

As previously mentioned, according to the integrating dimensions, this model is substantially maintained in subsequent validation works, like that by Kember and Leung (2011).

In the light of these former works, we consider it interesting to validate the instrument with the Spanish population, specifically students from three universities in the city of Valencia (Spain), two public ones and a private one, to determine to what extent it is maintained in this Spanish population (Kember & Leung, 2005b; Kember, Leung & Ma, 2007; Kember & Leung, 2009). This was our working objective.

Given the usefulness of the instrument and bearing in mind that we have no Spanish validation data, we believe this to be a relevant task. Despite having made a thorough search as possible, no works were found in the literature about validating the questionnaire with a Spanish population in either the Spanish-speaking or the English-speaking literature, or in other types, apart from the validation works conducted by the afore-cited authors over the years.

Method

Design

A test validation design was used (Crocker & Algina, 1986; Jornet & Suárez, 1996; Popham, 1990) to corroborate the instrument’s structure.

Participants

The sample was formed by 805 students from three universities in the city of Valencia: two public ones (Universidad de Valencia-UV, 69.2%, and Universidad Politécnica de Valencia-UPV, 15.8%) and one private one (Universidad Católica de Valencia-UCV, 15.0%). Of the 805 students, 540 were studying Education degrees (67.1%) (Social Education, Pedagogy and Teacher Training), 140 were studying Health Degrees (17.4%) (Medicine and Nursing) and 125 were studying Engineering degrees (15.5%) (Chemical Engineering and Industrial Engineering); 498 were year-1 students (61.9%), 223 were year-2 students (27.7%) and 84 were year-3 students (10.4%). Finally, 250 were males (31%) and 555 were females (69%).

Non-probabilistic purposive sampling was used as students were selected for being the students of the teachers who participated in the research and by selection criterion taken to select students was that they belonged to three main areas: Education, Health and Engineering. This was because the main research objective was to compare, in accordance with the cited areas, with those worked with the impact of the methods that centre on students’ learning and their performance. This was the main basis of the research project.

Instruments

The Kember and Leung (2009) version of the SEQ was used, which had 35 items and assesses the 15 aforementioned capabilities/variables: seven for students and eight for the teaching-learning environment. The SEQ uses a Likert-type scale with five response options, which range from Completely Disagree to Completely Agree. The questionnaire is included as Annexe I at the end of the work. The questionnaire was translated into Spanish by three research team members who mastered English, and their translations were compared to agree about which was to be revised by a native translator expert in translating and revising articles in the psychopedagogic field. This allowed the definitive translation to be agreed on.

Procedure

Students, who participated in a broader research work, answered the questionnaire when the teaching of a subject began in the first 4-month period, and they gave informed consent and used this platform: https://poliformat.upv.es/portal. To complete this instrument and the others employed in the research, the UV and UCV students were accompanied by their teachers during class hours to the computer rooms in their faculties, while those from the UPV answered them on their own.

Data analysis

Data were processed using the LISREL 8.80 programme (Jöreskog, Sörbom, Du Toit & Du Toit, 1999) to compare the structural model. The model’s estimations, used to confirm the scale’s dimensionality, were made by the Robust Maximum Likelihood Method given the multivariate non-normality of the used dimensions (χ2 = 1582.596, p < .001). To assess the model’s fit, χ2 SB was used by following the procedure proposed by Satorra-Bentler (Satorra & Bentler, 1994) given the selected procedure, and other indicators were employed according to the recommendations made on this matter (Byrne, 2006; Kline, 2005; Hair, Black, Babib & Anderson, 2010). With these recommendations, the route mean square error of approximation (RMSEA) was selected, which considered a good fit for a value below 0.05, along with its 90% confidence interval. Moreover, incremental fit indices, the Comparative Fit Index (CFI) and the Goodness-of-Fit Index (GFI) were obtained in which a good fit was considered when values equalled or exceeded 0.95 and 0.90, respectively. The Standardised Root Mean Residual (SRMR) was included, whose values which equalled or went below 0.08 indicated an acceptable fit. To assess the reliability of the dimensions, the indicators proposed by Raykov were used (2001, 2004) for the Structural Analysis Model, and information was added for Cronbach’s alpha coefficient (1951) to be particularly compared with the results offered by Kember & Leung (2009) in their proposal.

Results

Firstly, we describe the descriptive statistics and realibility of plots. Secondly, we provide details of the structural model to be validated.

Descriptive statistics and reliability of plots

In the descriptive statistics, the mean of the different plots (see Table 2) reveals students’ agreement, which ranges from 2.96 and 3.54. The lowest mean values appear in Coherence of Curruculum and Active Learning, with a mean value around 3. The highest mean values are found for the dimensions Self-managed Learning, Problem Solving, the Teacher-Students Relationship and Interpersonal Skills. The relative variability that the different plots present is medium-high, which represents a certain heterogeneity in students’ responses.

As for the relationship among the different plots, Pearson’s correlation in almost all the links among plots is higher than .50, which indicates a large effect size (Cohen, Cohen, West & Aiken, 2003). The relationship between Cooperative Learning and the other plots, except for the Relationship with Other Students, presents lower values in Pearson’s correlation, with a small effect size (Cohen et al., 2003). It is also necessary to consider that the plot Coherence of Curriculum shows median values in the relationship with other variables, with a medium effect size (Cohen et al., 2003).

Table 2: Descriptive statistics of the plots and the correlations matrix among the plots that the model includes. 

**The correlation is significant at 0.01 (bilateral).

Finally, Table 3 includes the reliability of the plots to be considered. Cronbach’s α coefficient ranges from .75 to .89, which is a suitable value.

Table 3: The Reliability indicators of the model’s plots (α, omega, ρxx’) and those offered by Kember and Leung (2009). 

Structural Model

The proposed model clearly obtained a good fit (see Table 4). The χ2 SB was significant, in accordance with the previously indicated considerations, but is not considered a relevant criterion to assess it. The RMSEA indicated an excellent fit with a very narrow oscillation range within which likelihood was 92%. SRMR, CFI and GFI also presented excellent fits.

Table 4: The fit indicators of the proposed model and of the study by Kember and Leung (2009). 

In the measuring model, the saturations of the different plots (see Figure 2) on their respective dimensions were significant (p < .01), and their value exceeded .79, except for the dimensions Coherence of Curriculum (.57) in the Teaching dimension, and Cooperative Learning (.36) in the Student-Student Relationship dimension.

The average variance extracted (AVE) was between 64.1% and 78.3%, except for the Student-Student Relationship dimension, which was 40.9%, and was slightly lower than the 50% value recommended by Hair et al. (2010). Therefore, convergent validity was suitable.

The reliability of the dimensions (see Table 5) was between .84 and .91 in relation to the omega coefficient, except for the Student-Student Relationship dimension that had a value of .55. Cronbach’s α was between .70 and .93 for all the dimensions. The ρxx’ estimations (Raykov, 2001, 2004) were calculated with the structural model results. The rho-based estimations were more conservative, which is not unusual and indicates a generally good level for the model’s dimensions (Teaching, Teacher-Students Relationship, Student-Student Relationship, Intellectual and Working Together), except for the Student-Student Relationship, which presented metric problems related with the Cooperative Learning plot.

Therefore, the different plots were relevant and consistent for structuring the model’s five latent variables and point out a suitable significant fit to explain the structural model.

Table 5: The Reliability indicators of the model’s structural dimensions (α, ρxx’) and those of Kember and Leung (2009). 

1. AVE: Average Variance Extracted2. Values estimated using the data in the work

The structural model (see Figure 2) presented significant estimations in all the considered parameters (p ≤ 0.01 or even higher). In fact the relationships between dimensions at the end (right) of the model all showed considerable significance (p ≤ 0.001).

In the part of the background dimensions in the model (Teaching, Teacher-Students Relationship and Student-Student Relationship) we can see that the relationships strongly related, and that with the lowest value is the link between the Teacher-Students Relationship and the Student-Student Relationship.

The Working Together mediation dimension was strongly influenced by the dimensions Teaching and the Student-Student Relationship, which was more consistent in the latter.

NB. In the asymmetric relations *** p ≤ 0.001; ** p ≤ 0.01; * p ≤ 0.05. All the other model’s parameters were significant p ≤ 0.01

Figure 2: Structural model in the Spanish sample of university students. 

The Intellectual Capability dimension that resulted from the model was influenced by both the Teaching dimension and Working Together, and the latter was better related with the Intellectual dimension.

Discussion

The objective of this work was to validate the SEQ with a Spanish population by verifying whether the dimensional structure defended by the authors (Kember & Leung, 2005b; Kember, Leung & Ma, 2007; Kember & Leung, 2009) was sustained with the data collected from the Spanish sample. The obtained results indicated that our research results conveniently sustained the structure of the model’s dimensions and its relations.

Firstly, the mean values of the 15 plots (Table 2) were slightly lower in the Spanish sample than in the studies of Kember and Leung (2005b) and by Kember, Leung and Ma (2007), except for the plots Interpersonal skills, Active Learning, the Teacher-Students Relationship and the Relationship with Other Students, which obtained a slightly higher mean value in the Spanish sample.

The estimation obtained in the plots for Reliability was quite acceptable in the present study. If compared (see Table 3) with the data from the authors’ other studies (Kember & Leung, 2005a; Kember, Leung & Ma 2007; Kember & Leung, 2009; Kember & Leung, 2011), they presented generally higher levels as far as Cronbach alpha coefficient was concerned. The ρxx’ estimations (Raykov, 2001, 2004) were also good, save those referred to for the plots Relationship with Other Students, Cooperative Learning and Coherence of Curriculum, which required subsequent tests with other samples to obtain more compelling evidence. Therefore, we can state that the plots can be used for making a diagnosis and for decision making in situations in which university teaching is consistently assess.

Secondly, the presented model’s fit indicators were good, and even better than for those reported in the previous cited studies that presented this model (Kember & Leung, 2009) (see Table 4).

Regarding the measuring model, the plots suitably defined the three Teaching-Learning Environment dimensions: Teaching, the Teacher-Student Relationship and the Student-Student Relationship; as well as the two dimensions in the capabilities area: Intellectual Capabilities and Working Together. If we compare the saturations found in the Spanish sample with those indicated in the study by Kember and Leung (2009), we find that they presented a higher value, except for the plots Coherence of Curriculum and Cooperative Learning, which presented lower saturations.

The omega reliability coefficient in the dimensions Teaching, the Teacher-Students Relationship, Intellectual Capabilities and Working Together were suitable and their values were higher than those of Kember and Leung (2009) (see Table 5). Nonetheless, the dimension Student-Student Relationship, with an acceptable value in the Spanish sample, was lower than in the research by Kember and Leung (2009), but did not reach the reference value.

Regarding the structural model, the model of the dimensional relation of the authors was confirmed (Kember & Leung, 2005b; Kember, Leung & Ma 2007; Kember & Leung, 2009), although the relationship between Teaching and Working Together was significant in the Spanish sample, as presented in other works by the cited authors (Kember & Leung, 2005a; Kember, Leung & Ma 2007), but not in the model of Kember and Leung de 2009. The values obtained for the found relationships were higher than those in the other studies of the cited authors (Kember & Leung, 2005a; Kember, Leung & Ma 2007; Kember & Leung, 2009), except for the Teaching and Intellectual Capability relationship, which was lower.

All these results allow us to state that a suitable instrument is available to diagnose and assess the development of both university students’ capabilities in the Spanish population, and the teacher’s capabilities to provide teaching and an assessment to generate an enriched and constructive learning environment for students by bearing in mind the interactions that derive from this second model’s plot in students’ capabilities. Using this instrument can provide teachers with relevant clues to assess to what extent their design of the Teaching-Learning Environment favours students developing capabilities and refining their methodological proposals to optimise them.

One limitation of this work was to use non-probabilistic purposive sampling; although the sample cannot be considered to represent the Spanish population, the variability of Degrees mitigated this possible limitation. All in all in future research works, the model should be compared in other larger samples. Finally, the model could be extended with other constructs, such as learning strategies or learning approaches, to better understand university students’ learning processes.

Acknowledgements

This work has been possible thanks to the financing for the project entitled "Learning-centered methodologies at the University. Design, implementation and evaluation", paid by the Spanish Ministry of Economy and Competitiveness. National Call for Aid to finance R&D Projects, 6th National R&D&I Plan 2008-2011, of 2011 (2013-2015) (Code EDU2012-32725).

References

Barrie, S. (2006). Understanding what we mean by the generic attributes of graduates. Higher Education, 51(2), 215-241. [ Links ]

Byrne, B.M. (2006). Structural equation modelling with EQS. Basic Concepts, Aplications and Programing. London: Lawrence Erlbaum Associates. Segunda edición. [ Links ]

Cajide, J. (1994). Análisis factorial confirmatorio de las características de la calidad docente universitaria, Bordón, 46 (4), 389-405. [ Links ]

Cohen, J., Cohen, P., West, S. G. & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. London: Routledge. [ Links ]

Coll, C. (2000). La concepción constructivista como instrumento para el análisis de las prácticas educativas escolares. En C. Coll, J. Palacios y A. Marchesi, Desarrollo psicológico y educación 2. Psicología de la educación escolar (pp. 29-64). Madrid: Alianza Editorial. [ Links ]

Coll, C. (2008a). Concepciones y tendencias actuales en psicología de la educación. En C. Coll (Coord), Psicología de la instrucción: la enseñanza y el aprendizaje en la educación secundaria (pp. 16-44). Barcelona: ICE/Horsori. [ Links ]

Coll, C. (2008b). La concepción constructivista como instrumento para el análisis de las prácticas educativas escolares. En C. Coll (Coord), Psicología de la instrucción: la enseñanza y el aprendizaje en la educación secundaria (pp. 16-44). Barcelona: ICE/Horsori. [ Links ]

Crocker, J.C. & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart and Winston. [ Links ]

Cronbach, L. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16 (3), 297-334. doi: 10.1007/BF02310555 [ Links ]

Doménech Betoret, F. (2007). Psicología de la educación e instrucción: su aplicación al contexto de la clase. Castellón: Universitat Jaume I. [ Links ]

Ertmer, P.A. & Newby, T.J. (2013). Behaviorism, cognitivism, constructivism: comparing critical features from an instructional design perspective. Internatial Improvement Quarterly, 6 (4), 43-71. doi: 10.1002/piq.21143 [ Links ]

Feldman, K. A. (1996). Identifying exemplary teaching: Using data from course and teacher evaluations. New Directions for Teaching and Learning, 65, 41-50. [ Links ]

García Ramos, J.M. (1997a). Valoración de la competencia docente del profesor universitario: una aproximación empírica. Revista Complutense de Educación, 8 (2), 81-108. [ Links ]

García Ramos, J.M. (1997b). Análisis factorial confirmatorio de la Valoración del Constructo Competencia Docente del Profesor Universitario. Bordón, 49 (4), 361-391. [ Links ]

García Ramos, J.M. (1998). Análisis de estructuras de covarianza en el estudio de la Competencia Docente del Profesor Universitario. Revista de Investigación Educativa, 16(1), 155-184. [ Links ]

García, L.A., Díaz, C. E., Ramírez, J.M. & Castro, J.J. (2008). Las competencias para el empleo en los titulados universitarios. Tenerife: Universidad de La Laguna. OPSIL (Observatorio Permanente para el Seguimiento de la Inserción Laboral). [ Links ]

García Espejo, I. & Ibáñez Pascual, M. (2006). Competencias para el empleo. Demandas de las empresas y medición de los desajustes. Revista Internacional de Sociología, LXIV (43), 139-168. [ Links ]

Gargallo, B. (2008): Estilos de docencia y evaluación de los profesores universitarios y su influencia sobre los modos de aprender de sus estudiantes. Revista Española de Pedagogía, 241, 425-445. [ Links ]

Gargallo, B., Garfella, P.R., Pérez, C. & Fernández, A. (2010). Modelos de enseñanza y aprendizaje. Ponencia presentada en el XXIX Seminario Interuniversitario de Teoría de la Educación "Formación y participación de los estudiantes en la universidad". Madrid, Universidad Complutense, Noviembre. http://www.redsite.es/docu/29site/ponencia3.pdfLinks ]

Gargallo, B. Sánchez Peris, F. , Ros, C. & Ferreras, A. (2010). Estilos docentes de los profesores universitarios. La percepción de los alumnos de los buenos profesores. Revista Iberoamericana de Educación, 51 (4), 1-16. [ Links ]

González-Pienda, J.A. (1999). El estudiante: variables personales. En J. Beltrán y C. Genovard, Psicología de la Instrucción I. Variables y procesos básicos (pp. 148-191). Madrid: Síntesis Psicología. [ Links ]

Gow, L. & Kember, D. (1993). Conceptions of teaching and their relationship to student learning. British Journal of Educational Psychology, 63, 20-33. doi: 10.1111/j.2044-8279.1993.tb01039.x [ Links ]

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate Data Analysis (Seventh edition). Prentice Hall: Upper Saddle River. [ Links ]

Ibáñez-Martín, J.A. (1990). Dimensiones de la competencia profesional del profesor de Universidad. Revista Española de Pedagogía, 186, 239-280. [ Links ]

Ibáñez-Martín, J.A. (2001). El profesorado de universidad del tercer milenio. El nuevo horizonte de sus funciones y responsabilidades. Revista Española de Pedagogía, 220, 441-466. [ Links ]

Jöreskog, K. G., Sörbom, D., Du Toit, S. & Du Toit, M. (1999). LISREL 8: New statistical features. Chicago: Scientific Software International. [ Links ]

Jornet, J. M. & Suárez, J. M. (1996). Pruebas estandarizadas y evaluación del rendimiento: usos y características métricas. Revista de Investigación Educativa, 14 (2), 141-163. [ Links ]

Kember, D. & Gow, L. (1994). Orientations to teaching and their effect on the quality of student learning. Journal of Higher Education, 65(1), 58-74. doi: 10.2307/2943877 [ Links ]

Kember, D. & Leung, D. (2005a). The influence of active learning experiences on the development of graduate capabilities, Studies in Higher Education, 30(2), 155-170. doi: 10.1080/03075070500043127 [ Links ]

Kember, D. & Leung, D. (2005b). The influence of the teaching and learning environment on the development of generic capabilities needed for a knowledge-based society. Learning Environments Research, 8, 245-266. doi: 10.1007/s10984-005-1566-5 [ Links ]

Kember, D. & Leung, D. (2009). Development of a questionnaire for assessing students’ perceptions of the teaching and learning environment and its use in quality assurance. Learning Environments Research, 12, 15-29. [ Links ]

Kember, D. & Leung, D. (2011). Disciplinary differences in student ratings of teaching quality. Research in Higher Education, 52, 278-299. doi: 10.1007/s11162-010-9194-z [ Links ]

Kember, D., Leung, D. & Ma, R. (2007). Characterizing learning environments capable of nurturing generic capabilities in higher education. Research in Higher Education, 2, 48(5), 609-632. doi: 10.1007/s11162-006-9037-0 [ Links ]

Kintsch, E., Franzke, M. & Kintsch, W. (1996). Principles of learning in multimedia educational systems. Tech Rep. No. 96-01. Boulder, CO: University of Colorado, Institute of Cognitive Science. [ Links ]

Kline, R. B. (2005). Principles and Practice of Structural Equation Modeling (2nd ed.). New York: Guilford. [ Links ]

Knight, P.T. (2005). El profesorado de educación superior. Formación para la excelencia. Madrid: Narcea. [ Links ]

Longworth, N., & Davies, W. K. (1996). Lifelong learning. London: Kogan Page. [ Links ]

Marsh, H. W. (1987). Students’ evaluations of university teaching: research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11, 253-388. [ Links ]

Marsh, H. W. (2007). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. In R. P. Perry& J. C. Smart(Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319-383). New York: Springer. [ Links ]

Mas, O. (2012). Las competencias del docente universitario: la percepción del alumno, de los expertos y del propio protagonista. REDU. Revista de Docencia Universitaria, 10(2), 299-318. [ Links ]

Mayer, R.E. (2010). Aprendizaje e instrucción. Madrid: Alianza Editorial. [ Links ]

Monereo, C. (1990). Las estrategias de aprendizaje en la educación formal: enseñar a pensar y sobre el pensar. Infancia y Aprendizaje, 50, 3-25. [ Links ]

Monereo, C. & Domínguez, C. (2014). La identidad docente de los profesores universitarios competentes. Educación XXI, 17 (2), 83-104. [ Links ]

Monereo, C. & Pozo, J.I. (2003). La universidad ante la nueva cultura educativa. Madrid: Síntesis. [ Links ]

OCDE (2005). La definición y selección de competencias clave. Recuperado de http://comclave.educarex.es/pluginfile.php/130/mod_resource/content/3/DESECO.pdf">http://comclave.educarex.es/pluginfile.php/130/mod_resource/content/3/DESECO.pdfLinks ]

Pascarella, E.T. & Terenzini, P.T. (1991). How college affects sutdents: findings and insights form twenty years of research. San Francisco: CA, Jossey-Bass. [ Links ]

Popham, J. (1990). Modern educational measurement. Boston, MA: Allyn and Bacon. [ Links ]

Ramsden, P. (1991). A performance indicator of teaching quality in higher education: The Course Experience Questionnaire. Studies in Higher Education, 16, 129-150. doi: 10.1080/03075079112331382944 [ Links ]

Raykov, T. (2001). Testing multivariable covariance structure and means hypotheses via structural equation modeling. Structural Equation Modeling, 2, 224-257. doi: 10.1207/S15328007SEM0802_4 [ Links ]

Raykov, T. (2004). Point and interval estimation of reliability for multiple-component measuring instruments via linear constraint covariance structure modeling. Structural Equation Modeling, 11, 342-356. doi: 10.1207/s15328007sem1103_3 [ Links ]

Rodríguez Espinar, S. (1993). Experiencias españolas de evaluación de la enseñanza universitaria y nuevas perspectivas. En Actas de las III Jornadas Nacionales de Didáctica Universitaria (pp. 111-132). Las Palmas de Gran Canaria: Universidad de Las Palmas de Gran Canaria. [ Links ]

Sampascual, G. (2010). Psicología de la educación Tomo I. Madrid: UNED [ Links ]

Satorra, A., & Bentler, EM. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. von Eye y C.C. Clogg (Eds.), Latent variables analysis: Applications for developmental research (pp. 399-419). Thousand Oaks, CA: Sage. [ Links ]

Tait, H., & Godfrey, H. (1999). Defining and assessing competence in generic skills. Quality in Higher Education, 5, 245-253. [ Links ]

Tejedor, F.J. (1993). Experiencias españolas de evaluación de la enseñanza universitaria. En Actas de las III Jornadas Nacionales de Didáctica Universitaria (pp. 85-109). Las Palmas de Gran Canaria: Universidad de Las Palmas de Gran Canaria. [ Links ]

Torra, I., Márquez, Mª D., Pagès, T., Solà, P., García, R., Molina, F. et al. (2013). Retos institucionales de la formación del profesorado universitario. REDU. Revista de Docencia Universitaria 11 (1). 285-309, doi:https://doi.org/10.4995/redu.2013.5601. [ Links ]

Trigwell, K. & Prosser, M. (2004). Development and Use of the Approaches to Teaching Inventory. Educational Psychology Review, 16 (4), 409-424. doi: 10.1007/s10648-004-0007-9 [ Links ]

Trillo, F. & Méndez, R. Mª. (1999). Modelos de enseñanza de los profesores y enfoques de aprendizaje de los estudiantes: un estudio sobre su relación en la Universidad de Santiago de Compostela. Adaxe, 14-15, 131-147. [ Links ]

Tuning (2009). Una introducción a Tuning Educational Structures in Europe; la contribución de las universidades al proceso de Bolonia. Recuperado de http://www.unideusto.org/tuningeu/images/stories/documents/General_Brochure_Spanish_version.pdfLinks ]

Vidal-Abarca, E. (2010). Aprender y enseñar: una mirada desde la Psicología. En E. Vidal-Abarca , R. García Ros y F. Pérez González , (eds.), Aprendizaje y desarrollo de la personalidad (pp. 19-43). Madrid: Alianza Editorial. [ Links ]

Villar Angulo, L.M. & Alegre de la Rosa, O.M. (2004). Manual para la excelencia en la enseñanza superior. Madrid: McGrawHill. [ Links ]

Wilson Lizzio, A., & Ramsden P. , (1997). The development, validation and application of the Course Experience Questionnaire. Studies in Higher Education, 11 (1), 33-53. [ Links ]

Zabalza, M.A. (2003). Competencias docentes del profesorado universitario. Calidad y desarrollo profesional. Madrid: Narcea. [ Links ]

Zabalza, M.A. (2007). Planes de formación del profesorado universitario. Encuentro ANECA sobre La evaluación del profesorado dentro de los Sistemas de Garantía de Calidad de las Instituciones Universitarias. http://www.aneca.es/servicios/docs/burgos07_09_zabalza.pdfLinks ]

Zimmerman, B.J. (2002). Becoming a self-regulated learner: an overview. Theory into Practice, 41, 64-70. doi: 10.1207/s15430421tip4102_2 [ Links ]

Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1), 166-183. doi: 10.3102/0002831207312909. [ Links ]

This work has been possible thanks to the financing for the project entitled "Learning-centered methodologies at the University. Design, implementation and evaluation", paid by the Spanish Ministry of Economy and Competitiveness. National Call for Aid to finance R&D Projects, 6th National R&D&I Plan 2008-2011, of 2011 (2013-2015) (Code EDU2012-32725).

1The three terms have often been used as if interchangeable or synonyms, but not strictly so. In the psychopedagogic field, the term skill, understood as the dexterity or facility to perform activities or tasks, tends to be interpreted as being more restrictive than either capability (understood as a series of resources, conditions, qualities or attitudes an individual has to perform a task or misión; thus it implies the potentiality to do something), or competence (which implies a combination of knowledge, skills, attitudes and values that allow one to act efficiently and to adapt to the requirements/demands of a given profession, learning, etc.). This implies putting into action available potentialities, although the competence concept has not always been univocal. In the DeSeCo project, competences are understood as “the skill to face complex demands” (OECD, 2005: 3) by resorting to psychosocial resources (including dexterity and attitudes) in a given context. Thus competence is interpreted as a special ability of a higher order. In the Tuning project, “competences represent a dynamic combination of cognitive and metacognitive capabilities, knowledge and understanding, which are interpersonal, intellectual, practical and ethical values” (Tuning, 2009: 3). Thus competences are understood as capabilities and a combination of capabilities. Indeed in this project, which has been a reference for designing university degrees in the Spanish context, the list of skills includes capabilities as “examples of the skills” to be included in degrees (along with the capability to analyse and summarise, critical capability, etc.). For us, a plausible interpretation would be to consider competences in a higher hierarchy above the abilities and capabilities that they would integrate. However, the terms capability and competence are frequently used in an interchangeable way in the literature, as is sometimes the term skilll or ability. In the Anglo-Saxon world, the term capability is often used to refer to skills which, in our context, and specifically since the Bologne process has been developed, we call competence.

Received: July 07, 2017; Accepted: January 08, 2018; Accepted: February 16, 2018

* Correspondence address [Dirección para correspondencia]: Bernardo Gargallo. Departamento de Teoría de la Educación. Facultad de Filosofía y Ciencias de la Educación. Universidad de Valencia. Avda. Blasco Ibáñez, 30. 46010, Valencia (Spain) Email: Bernardo.gargallo@uv.es

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License