Ambulatory Assessment

Ambulatory Assessment (AA) can be defined as a systematic and structured procedure for studying people's behaviour (affect, cognition, mental states, etc.) in their natural environment and in real time (or near real time), at multiple moments in time, usually using an electronic device, such as a personal digital assistant (PDA) or smartphone (^{Fonseca-Pedrero et al., 2022}; ^{Myin-Germeys & Kuppens, 2021}; ^{Trull & Ebner-Priemer, 2013}). AA uses a variety of data sources i.e., multimodal assessment (e.g., psycho-physiological, biological, self-reported, and behavioural) and represents a conceptual and methodological umbrella that includes experience sampling methodology (ESM), ecological momentary assessment (EMA) and momentary assessment (^{Stone & Shiffman, 1994}).

AA attempts to overcome some of the problems associated with the traditional assessment paradigm such as the lack of ecological validity (less generalizability) and the biases associated with retrospective assessments. At the same time, AA allows researchers to investigate within-person processes and patterns of variation over time, assess context-specific relationships, and deliver feedback in real time (^{Ebner-Priemer & Trull, 2009}; ^{Myin-Germeys & Kuppens, 2021}).

AA has grown exponentially in recent years, gaining importance and popularity in different areas of psychology (^{Heron et al., 2017}; ^{Myin-Germeys et al., 2018}; ^{van Roekel et al., 2019}). Developments and studies have been conducted in the field of clinical psychology, in areas such as anxiety and depression (^{Hall et al., 2021}), psychosis (^{Bell et al., 2017}), and suicidal behaviour (^{Sedano-Capdevila et al., 2021}). Particularly noteworthy are the studies on the mechanisms and dynamics of symptoms, the prediction of the recurrence or onset of symptoms, the monitoring of treatment effects, and the prediction of treatment success (^{Trull & Ebner-Priemer, 2013}).

Due to the novelty and growth in popularity of AA, some general reporting guidelines have been published (^{Heron et al., 2017}; ^{Trull & Ebner-Priemer, 2020}). According to ^{Trull and Ebner-Priemer (2020)}, when carrying out a study under this new paradigm, the following should always be reported on: (a) the sample size and selection, (b) sampling design, (c) selection and reporting of measures, (d) devices and software used, (e) compliance, (f) participant training, monitoring and remuneration, and (g) data management and analysis.

Social Network-Based Assessment in the Organizational Realm

In the context of the war for talent (^{Frasca & Edwards, 2017}), technology has emerged as a key element in the field of human resource management (HRM) (^{Ryan & Derous, 2019}). Social network websites (SNW) have become one of the main technology tools used by professionals in recruitment and selection processes (^{Nikolaou, 2014}; ^{Woods et al, 2020}). SNW profiles allow HRM professionals to gather information about applicants' knowledge, skills, abilities and other characteristics and examine the degree to which applicants' qualifications are aligned with the job requirements or fit with the organizational culture (^{Bangerter et al., 2012}). SNWs provide more information than traditional methods (^{Zide et al., 2014}) and at a lower cost (^{Nikolaou, 2014}). A SNW is characterized by allowing the user to: (a) define a profile within a bounded system, (b) articulate a list of contacts with whom to share information, and (c) view and browse their own list and those of other users to identify opportunities for connection and contact (^{Boyd & Ellison, 2007}). For instance, as ^{Collmus et al. (2016)} point out, the information contained in a LinkedIn profile includes features of traditional résumés, reference checks, and recommendation letters.

HRM professionals do not only use SNWs such as LinkedIn to identify and contact potential candidates; they also use them to make selection decisions. To do this, they make inferences about the degree to which applicants will fit the job profile and the organization (^{Kluemper et al., 2012}; ^{Roulin & Bangerter, 2013}). Thus, from viewing LinkedIn profiles, professionals make inferences about the applicants' personality and competencies, and make predictions about their future performance (^{Van Iddekinge et al., 2016}). A key issue is that this decision-making process can be accompanied by deficits in the reliability of measures, and a lack of validity due to the scarcity of associated evidence. Research has attempted to shed light on how professionals use this information to make inferences and investigate the psychometric properties of LinkedIn as a selection tool.

Building on the Realistic Accuracy Model (RAM; ^{Funder, 2012}) and the Signaling Theory (^{Donath, 2008}; ^{Spence, 1973}), ^{Roulin and Levashina (2019)} found that the reliability and validity of the inferences about candidates' skills, personality and cognitive ability are different depending on the characteristics assessed. Inferences regarding skills and personality have lower inter-rater reliability than either cognitive ability inferences (ICC = .60) or hiring recommendations (ICC = .67). Temporal reliability is similar for all the characteristics (correlations between .41 and .66). Further differences exist depending on the ability or inferred personality trait (skills and traits such as conflict management and conscientiousness are less reliable while estimations of communication skills and agreeableness are more reliable). Regarding the convergent validity of these inferences based on self-report measures, the findings seem to make it clear that personality traits are not inferred accurately, and that the only skills that are inferred more accurately are leadership, planning and communication skills (correlations between .23 and .27). Cognitive ability also seems to be inferred accurately. Finally, regarding predictive validity, the global analysis of the profiles seems to correlate moderately (between .20 and .25) with the development of a successful professional career.

Reliability and validity seem to improve through two strategies: the use of itemized signals for profile information (^{Fernández et al., 2021}; ^{Roulin & Levashina, 2019}), and the use of clusters of LinkedIn information. By using the latter strategy, ^{Aguado et al. (2019)} established that the information on LinkedIn profiles can be grouped into four large dimensions (LinkedIn Big Four): (a) breadth of professional experience, (b) social capital, (c) interest in updating knowledge, and (d) breadth of non-professional information. These four dimensions have proven useful in predicting employee outcomes such as business and management potential, absenteeism, and productivity. Also, they seem to constitute valid signals of the generic competencies of users (^{Andrés et al., 2022}). In addition, when standardized tools such as rubrics are used to assess these dimensions, the inter-rater and temporal reliability increases substantially (^{Andrés et al., 2022}).

Finally, research seems to suggest that the use of LinkedIn does not generate an adverse impact in terms of gender, and only a small to moderate adverse impact when ethnicity (white vs. non-white) is examined (^{Roulin & Levashina, 2019}).

Gamification

Technology has also facilitated interactive assessment through computerized games, a field which has made extraordinary advances in recent years. In gamification, the subject being evaluated interacts with the computer game with a view to achieving a series of specific results. The subject's behaviour throughout this interaction is automatically assessed and an estimation of the psychological characteristic being measured is produced. It is not exactly new, it is what ^{Cattell called T-data (Cattell & Warburton, 1967}). In fact, within the area of Objective Personality Tests (e.g. ^{Ortner & Proyer, 2015}) different tools have been developed to assess risk tendency (^{Lejuez et al., 2002}; ^{Aguado et al., 2011}), impulsivity (^{Elliot et al., 1996}), heuristic thinking (^{Jasper & Ortner, 2014}), achievement motivation (^{Ziegler et al., 2010}), and Attention-Deficit Hyperactivity Disorder (^{Delgado-Gómez et al., 2020}). Despite the advantages of these tools in the observation of behaviours, compared to self-assessment, and in terms of resistance to faking (^{Ortner & Proyer, 2015}), the convergent validity results point to the need to delve further into their relationship with the gold standard for measuring personality, i.e. self-reports.

Adopting a different approach, some authors have developed the idea of using serious games for assessment purposes (e.g., ^{Kato & Klerk, 2017}). Serious games are games that are used for a purpose other than entertainment, usually for skill training and assessment (e.g., ^{Caballero-Hernández et al., 2017}). Serious games are characterized by using game design frameworks (e.g., rewards, objectives, simulated scenarios) together with advanced technological support (e.g., online interactive software, immersive 3D environments, virtual reality). From a psychometric perspective, the use of serious games is based on the hypothesis that they have the potential to offer more valid measures than traditional approaches. This is possible because: (a) they offer evaluative scenarios with greater ecological validity, (b) serious games make it possible to carry out behaviour-based assessment instead of relying on self-reports, and (c) they allow the use of technological advances in data collection and analysis, as well as in the presentation of information (^{Lumsden et al., 2016}).

However, despite its promising potential in assessment, the field of gamification has received little attention in the literature on psychometric and psychological assessment (^{Ryan & Ployhart, 2014}). As a result, gamification strategies have often not been subjected to the quality and rigorous standards of psychometric measurement (^{Landers, 2015}). Therefore, reliable theories need to be developed on the impact of different elements on game assessment results (^{Armstrong et al., 2016}).

Forced-Choice Testing and New Psychometric Models

Most of the self-report scales for the measurement of non-cognitive traits use a Likert scale format, in which a respondent indicates his or her level of agreement with a statement. However, even though meta-analysis studies support the predictive role of self-report scales in applied contexts (^{Judge et al., 2013}), it is well known that this format is sensitive to response biases such as social desirability or acquiescence bias (^{Kreitchmann et al., 2019}). In selection contexts, social desirability and faking increases applicant mean scores in the perceived desirable direction while reducing the reliability and variability of scores (^{Salgado, 2016}).

As an alternative, a forced-choice format requires the test taker to partially or totally rank a set of assertions into blocks of two or more statements, based on how well they feel described by them. Response biases such as acquiescence do not affect forced-choice tests, and, if the blocks are built with items that are well-matched in terms of social desirability, they should be less susceptible to the effects of faking. The following are typical formats (^{Hontangas, 2016}): (a) choose the item that best describes you from two statements, (b) choose the item that best describes you from more than two statements, (c) choose the item that best describes you and the item that least describes you, and (d) rank the alternatives according to the degree to which they describe you.

Forced-choice tests can be fully ipsative, quasi-ipsative or normative, depending on the degree of ipsativity of the obtained scores (^{Hicks, 1970}). Ipsativity means that a person's trait scores depend on his or her other trait scores and has been suggested to be the main drawback of forced-choice tests; it makes inter-individual comparisons difficult and biases typical psychometric analysis results (^{Hicks, 1970}). In the worst-case scenario, in fully ipsative measures, all test taker scores add up to a constant, distorting the internal structure of the test and the predictive validity of the scores. According to certain meta-analyses, quasi-ipsative assessments offer stronger predictive validity (^{Salgado et al., 2015}) and are less susceptible to faking (^{Martínez & Salgado, 2021}).

Forced-choice tests are designed to address the issue of social desirability in non-cognitive measures in selection tests. However, to generate reliable, faking-resistant, non-ipsative scores, optimum forced-choice test design and scoring are critical. First, the items of each block should be well-matched in terms of social desirability, based on expert consensus (^{Pavlov et al., 2021}). Second, the items should be paired optimally (e.g., ^{Kreitchmann et al., 2021}). Finally, the scoring could be performed using novel item response theory (IRT) models such as multi-unidimensional pairwise preference (MUPP; ^{Stark et al., 2005}) or Thurstone IRT (TIRT; ^{Brown & Maydeu-Olivares, 2011}). These models were developed to shape the probability of agreement with an item in a forced-choice block.

Recent advances include the use of genetic algorithms for the optimal assembly of forced-choice blocks (^{Kreitchmann et al., 2021}), the determination of the optimal selection rule in forced-choice Computerized Adaptive Tests (CATs), and the construction of CATs on the fly, in which blocks are assembled optimally at the time of administering the CAT (^{Kreitchmann et al., 2022}).

Furthermore, the use of novel multidimensional models is also showing promise. A bi-factor multidimensional structure is to be expected in many psychological domains, in which items measure both a general factor (e.g., the domain of extraversion) and a group factor (e.g., facet of gregariousness). ^{Nieto et al. (2018)} studied the performance of bi-factor CATs, demonstrating that the use of multidimensional models increases efficiency. Recent developments in the algorithms for the estimation of bi-factor structures (^{García-Garzón et al., 2020}) should make it simple to create forced-choice tests based on more realistic bi-factor structures.

In conclusion, while forced-choice tests have been around for a long time, it is only through methodological and technological advances that adequate modelling and scoring of forced-choice data, as well as the optimal design of forced-choice blocks and computerized adaptive applications, have become possible. Although it is challenging to construct an effective forced-choice test, well-designed IRT-based adaptive forced-choice tests will undoubtedly improve measurement in recruiting and selection processes.

Discussion

The paper has described advances in the field of digital technology-based psychological and educational measurement and assessment. Some of these developments address classical measurement problems such as response bias and modelling dimensionality, and can be seen as a natural progression in the history of psychometric research. In addition, the implementation of new technologies which bring know-how from the fields of engineering or computer science opens up a new horizon of possibilities. Social networks, the internet of things, the process data or the use of mobile devices are making it possible to use methodologies based on experience sampling, gaming and virtual environments. These could help to improve the ecological validity of assessment and better understand people's thoughts, feelings, and behaviours (^{Bogarín, 2018}; ^{Fonseca-Pedrero et al., 2021}; ^{López-Mora, 2021}; ^{Parsons, 2012}). It could be said that the social science research tradition built on ^{Cattell's (1966)} representation of the data cube is moving towards huge amounts of unstructured, high-dimensional data which need new data analytical approaches.

All of these new approaches can be seen as part of the broader field of Artificial Intelligence (AI), which is permeating science and society and includes specialized areas in computing, machine learning, natural language processing, computer vision, and robotics. There are many definitions of AI, but one of the most accepted is that proposed by ^{Russell and Norvig (2021)}. According to these authors, AI focuses on the study and construction of agents that do the right thing - the right thing being the goal set for the agent - and agent is defined as something that perceives its environment through sensors. In simple statistical terms the right thing to do could be the decision (estimate) that minimizes the loss function (^{Elosua, 2022}). This definition has been accepted by the European Union, which reformulates it as: Software that is developed using one or more of the following techniques and strategies and that can, for a given set of human-defined objectives, generate output information such as content, predictions, recommendations, or decisions that influence the environments with which it interacts (Machine learning strategies, including supervised, unsupervised, and reinforcement learning, that employ a wide variety of methods, including deep learning. Logic and knowledge-based strategies, especially knowledge representation, inductive programming (logic), knowledge bases, inference and deduction engines, expert and (symbolic) reasoning systems. Statistical strategies, Bayesian estimation, search and optimization methods). (^{European Commission, 2021}).

The use of natural language processing in automated personality assessment already offers high quality standards (^{Tausczik & Pennebaker 2010}; ^{Schwartz et al., 2013}; ^{Youyou et al., 2015}), automatic analysis of movement (^{Delgado-Gómez et al., 2016}). Analysis of voice characteristics has been successfully used in personality assessment (^{Mairesse et al., 2007}), in the prediction of sleep apnoea (^{Espinoza-Cuadros et al., 2016}), and for the detection of the early stages of Alzheimer's disease (^{Weiner et al., 2016}). The application of virtual scenarios and metaverse has generated assessment tools for the measurement of affective disorders, cognitive processes, spatial abilities, or attention and memory processes (e.g., ^{Knight & Titov, 2009}; ^{Law et al., 2006}; ^{Powers & Emmelkamp, 2008}).

However, applying this new technology in the fields of psychological and educational assessment brings with it a series of issues that need to be addressed (^{Elosua, 2022}): (a) a comprehensive analysis of the contribution of each innovation in terms of validity of the measurement needs to be carried out, (b) a debate on the ethical, legal, and social implications of the new methods of collecting/analysing data must take place, (c) the different fields of knowledge need to be coordinated, and (d) psychologists need to be trained in the new approaches to data analysis. In this regard, we understand that it is not easy to include sophisticated techniques during a psychometric program for undergraduate students, but we should make our students aware of the need for openness in order to adapt and take advantage of an environment in which analytical techniques in the field of artificial intelligence are progressing rapidly, and where interdisciplinary is a necessary condition to keep high scientific, and ethics standards.

Studies dealing with the associated ethical, scientific and social implications systematically report concerns about transparency, fairness, equity, and bias (^{Jobin et al., 2019}). In the field of AA, for instance, privacy and confidentiality issues have been reported (^{Fonseca-Pedrero et al., 2022}; ^{Palmier-Claus et al., 2011}). Given the right to privacy of job applicants, the use of recreational SNWs (such as Facebook or Instagram) in recruiting and selection processes may affect the legal liabilities of companies (^{Roulin & Levashina, 2019}). In addition, negative reactions by applicants to the use of recreational SNWs have already been reported (^{Aguado et al., 2019}; ^{Stoughton et al., 2015}; ^{Stoughton, 2016}). The lack of tradition and perceived intrusion in this particular field of psychological assessment is also linked to a lack of rigor and psychometric quality standards (^{Landers, 2015}) as well as to the need for sound theories on the impact of the different game elements on the measurements obtained (^{Armstrong et al., 2016}). There is also an open debate around the application of machine learning and/or other AI models to the field of assessment which alerts us of the consequences of a blind focus on maximizing predictive accuracy (^{Fokkema et al., 2022}). In this vein, some interesting attempts are being carried out to integrate analytic techniques derived from AI with theoretical psychometric research applied to learning and assessment; to date, computational psychometrics (CP) seems one of the most promising. CP basically explores analytical models for new types of data and studies how to integrate them to define components of teaching, learning, and assessment (^{Langenfeld, et.al., 2022}; ^{von Davier et al., 2021}).

Furthermore, despite the technological advances of recent years, an analysis of the professional practice indicates approximately 90% of tests are pen and paper (^{Santamaría & Sánchez-Sánchez, 2022}). These results show two clear trends: a traditional trend which is still reliant on the classical test model and a data-driven model built around the availability of data. The former is a deductive and theory-centered approach and the second is data-centric and exploratory; it applies machine-learning models that search for patterns and relationships which are used for classification and prediction purposes. With a view to advancing our knowledge and increasing our scientific productivity, it would be of great use to find a convergent area between the two models (^{Maass et al., 2018}). To do so, teamwork among different disciplines and training (^{Adjerid & Kelley, 2018}; ^{König et al., 2020}; ^{Oswald, 2020}) would be key to addressing the diversity and dynamism of this new age. What psychometrics have taught us about psychological measurement and what we already know about the science of psychology offer a very promising combination with which to face future challenges.