Test developers often rely on judgements made by subject matter experts (SMEs) as sources of evidence around assessment validity. In educational settings, experts may be consulted to evaluate the alignment of the items to the curricular standards and learning objectives, indicating whether an item is appropriate for assessing a given grade group, or has the desired depth of knowledge or cognitive complexity (e.g., ^{Bhola et al., 2003}; ^{Webb, 2007}). Similarly, in diagnostic assessments, experts are often consulted to determine the skills or abilities required for correctly solving each problem in a test (^{García et al., 2014}; ^{Nájera et al., 2021}; ^{Tatsuoka, 1983}). Likewise, for the assessment of non-cognitive domains, such as personality, motivation or leadership, SMEs are relied upon to evaluate the appropriateness of the items regarding various aspects. For instance, they may be asked to judge if different aspects of the constructs are properly represented in a test (e.g., ^{Polit & Beck, 2006}). In a similar fashion, SMEs may assess the degree to which each item is relevant to the construct, as well as the appropriateness of other aspects related to item wording, such as clarity, lack of ambiguity, and technical quality (^{Fitzpatrick, 1983}; ^{Mastaglia et al., 2003}; ^{Penfield & Giacobbi, 2004}). From a broad perspective, the efforts in ensuring the adequacy of the test content for the measured constructs partially constitute what is commonly called content validity, or, more correctly, validity evidence based on test content (^{American Educational Research Association [AERA] et al., 2014}).

Gathering validity evidence for test content involves determining the test's ability to accurately measure its intended purpose. Content-based validity evidence comprises four main aspects: domain definition, domain representation, domain relevance, and appropriateness of test construction procedures (^{Sireci, 1998b}). While the first refers to the definition of the measured domains, the others concern the test itself. Evaluating domain representation involves ensuring that the test accurately reflects content specifications, cognitive processes, etc. Inspecting domain relevance involves assessing the relevance of test items to the intended domain, as well as the overall relevance of the test to the assessment goals. Lastly, assessing construction appropriateness encompasses the extent to which process decisions are well-reasoned and quality controls are implemented (^{Sireci & Benítez, 2023}; ^{Sireci & Faulkner-Bond, 2014}).

Content-based validity evidence is foundational for assessment validity, as it establishes the appropriateness of the test content for the assessed construct. Other validity sources, like internal structure-based evidence, can be compromised if test content is inadequate. Poorly sampled items or irrelevant content can distort scores and interpretations (^{AERA et al., 2014}). Similarly, if test content has different effects on particular subgroups (e.g., based on socioeconomic status, race/ethnicity, or geography), scores may be biased (^{AERA et al., 2014}; ^{Gómez-Benito et al., 2018}). In this article, we specifically propose a way of improving the assessment of item relevance using SMEs, although the framework proposed here may be generalizable to other types of SME-based evidence around test content (e.g., social desirability, wording).

The assessment of domain representation and relevance is often conducted either through a matching task or a rating task (^{Sireci & Faulkner-Bond, 2014}). The former involves SMEs matching items to their closest domain and/or cognitive specifications (e.g., using Bloom’s taxonomy). The latter comprises SMEs using ordinal scales to rate (a) the item representativeness/relevance for their respective domains, or (b) the similarity between items to assess whether the items correctly follow the expected domain structure (e.g., ^{Li & Sireci, 2013}).

Once the tasks are completed, the responses must be summarized using validity indices. These indices provide information to facilitate test developers in their decisions about the inclusion/exclusion of certain items. For instance, a common index for matching tasks is the proportion of times an item is assigned to its correct domain (^{Sireci & Faulkner-Bond, 2014}). Low values indicate that the item has an unclear association with the expected domain. For rating tasks, several other summarization procedures or indices exist, such as ^{Rovinelli and Hambleton’s (1977)} item-objective congruence index or multidimension scaling of item similarities (^{Li & Sireci, 2013}). In alignment studies, evaluations often cover various aspects of assessments, like the extent to which target content areas and cognitive levels are accounted for in a test (^{Martone & Sireci, 2009}). To measure alignment, matrix comparison indices are often used, quantifying how well test elements (e.g., number of items categorized by depth of knowledge and assessment content) align with curriculum standards (^{Porter, 2002}).

Among the most commonly used and straightforward indicators in content validation, two stand out: the content validity index (CVI; ^{Martuza, 1977}) and Aiken's V index (^{Aiken, 1980}). The CVI can be calculated at both item (I-CVI) and scale (S-CVI) levels. The I-CVI reflects the proportion of experts who agree on the relevance/representativeness of each item, being computed as the proportion of experts that consider an item relevant/representative (i.e., endorsing the higher half of the rating scale). The S-CVI is defined as the proportion of items about which all judges completely agree upon their relevance. In turn, Aiken’s V consists of rescaling mean item evaluations to facilitate its interpretation:

where x- denotes the average rating for a given item, l is the lowest possible rating and K the highest rating option, thus providing a V value between 0 and 1. As it can be inferred from Equation 1, the I-CVI is a special case of Aiken’s V for dichotomized data (e.g., relevant vs. irrelevant).

Regardless of the type of task, gathering SME judgment-based is a delicate matter. Tasks often involve a small group of experts (often from 3 to 20 SMEs; ^{Almanasreh et al., 2019}; ^{McCoach et al., 2013}; ^{Rubio et al., 2003}), so each judgment has great implications. Specifically, the adequacy of the assessments largely relies upon three key aspects: (1) expert selection, (2) task comprehension and engagement, and (3) result summarization indices. First, aiming to ensure a proper sample of experts, SMEs are often selected based on expertise criteria (e.g., years of experience). Second, clear instructions must be given, including detailed definitions of the domains in the test blueprint. Lastly, the choice for the appropriate validity index should provide accurate scores, leading to proper decisions about the items.

In realistic scenarios, where SMEs may have different backgrounds, their expertise, engagement with the task, or their use of the response scales (e.g., leniency/severity), may vary even among highly qualified experts (^{Sireci, 1998a}). For instance, a more experienced rater may have a higher ability to discriminate between lowly and highly relevant items. Severe experts can show a higher tendency to use the lower end of the response scale than lenient experts. For instance, a score of 4 for an SME whose average rating is 2 may not be comparable to a score of 4 for an expert with average ratings of 3.

In applied settings, SME performance and the rating precision are often disregarded. Consequently, when summarizing the results with traditional indices (e.g., with CVI or Aiken’s V), these sources of measurement error are unaccounted for. In essence, despite the substantial advancements in measurement theory and methods in educational and psychological assessments (e.g., factor analysis, item response theory,), little has been translated into the context of SME-based validity assessment. In this sense, using statistical models to accommodate the SME variability may increase accuracy and generalizability of content validity scores, while also providing evidence on the appropriateness of the task (^{Rios & Wells, 2014}).

In this matter, the item response theory (IRT) framework is especially suitable for such purposes. In fact, previous studies have accounted for rater variability using IRT models in educational assessments involving multiple graders (e.g., ^{Lunz et al., 1994}; ^{Robitzsch and Steinfeld, 2018}; ^{Wu, 2017}). It has been found, for instance, that considering the variability of graders’ severity can have an impact on measurement accuracy and affect decisions regarding pass-fail outcomes (e.g., ^{Lunz et al., 1990}). In these applications, treating SMEs as measurement items in IRT models enables to estimate the discrimination and threshold parameters for each rater (e.g., if they correctly differentiate between relevant and irrelevant items, as well as their leniency/severity). This information provides evidence on the appropriateness of the experts, making it possible to detect suboptimal performance (e.g., SMEs that don’t fully understand the task). As a result, it allows for the estimation of the relevance/representativeness scores of each item while considering SMEs’ variability. Moreover, IRT provides a wide variety of evidence on the validity of the rating task, such as score reliability, standard errors, and goodness-of-fit indices. Accordingly, the goal of this article is to propose and illustrate the use of IRT modeling when summarizing content-related validity evidence from SMEs.

Within the IRT framework, the graded response model (GRM; ^{Samejima, 1968}) is a traditional polytomous response model. It is commonly applied in attitude and personality assessments, where Likert-type responses are frequent (e.g., ^{Collado et al., 2015}; ^{Kreitchmann et al., 2019}). The GRM models the probability of responding to each category of an item given the latent trait being measured. Specifically, the probability of endorsing category k or higher is a function of structural (i.e., item related) and incidental parameters (i.e., latent trait θ), as in Equation 2:

Specifically, it involves the estimation of K parameters per item, namely one slope parameter a, and K - 1 threshold parameters b, being K the number of response categories. Usually, slope (or discrimination) parameters (a) represent the ability of an item to discriminate between different levels of the latent trait. For readers familiarized with factor analysis, discrimination parameters are equivalent to factor loadings given Equation 3:

Threshold parameters represent the boundaries on the latent trait continuum where individuals are more likely to endorse category k or greater against previous categories. Each category, except the last one, has an associated threshold parameter, where b1<…<bK-1 .

Traditionally, the input data for estimating the GRM consists of a matrix of dimensions N respondents × J items. With SMEs ratings, however, the data matrix takes the form of I evaluated objects (e.g., items) × M raters. As a result, the GRM parameters assume distinct interpretations within this context. The slope parameter represents an SME’s ability to differentiate between items that are relevant and irrelevant in measuring the intended construct. The latent parameter θ reflects the specific feature being evaluated by the SMEs (e.g., item relevance). Finally, the b parameters are associated with the SME’s leniency or severity. They reflect the level of item representativeness or relevance demanded by an expert to assign category k or beyond with a probability of 0.50. These parameters offer insights into the individual judgments made by experts, indicating their willingness to endorse different response categories based on the item’s perceived relevance. To illustrate these parameters, the results for two SMEs in an expert rating task with four response categories are represented in Figure 1.

Figure 1. Operating Characteristics Curves for Two Raters. 

The rating task presented consisted in evaluating the relevance of 120 items for the measurement of the Conscientiousness domain. The response curves depicted in Figure 1, Panels A and B, represent item relevance ratings collected from two different raters: an expert rater with a Ph.D. in Psychology, and a non-expert rater with no degree in Psychology, respectively. As it can be observed, higher discrimination values correspond to steeper curves, indicating that smaller shifts in item relevance are required to move from category k - 1 to k. The threshold parameters were b_expert = {-0.83, -0.11, 0.54} and b_non-expert = {-1.52, -0.27, 1.12}. These parameters are defined in θ (standardized normal) metric. The more centralized values of the expert rater indicate a tendency to endorse more extreme responses. For instance, in the case of b₁, items with a standardized relevance score lower than -0.83 are more likely to be rated as 1 out of 4 by the expert rater. Conversely, for the non-expert rater, a relevance score as low as -1.52 would already suggest a higher probability of assigning a rating greater than 1.

The probability of endorsing each specific category k can then be computed by calculating the differences between consecutive cumulative probabilities. As in Equation 4:

This transformation allows to visualize probabilities curves for the selection of each category independently. In the content validation context with SMEs, these curves can be referred to as expert characteristic curves (ECC), which reflect the relationship between item relevance and the rater’s response probabilities. The corresponding ECCs of the previous example are shown in Figure 2.

Figure 2. Experts Characteristics Curves. 

The scores derived from the GRM ( θ^GRM ) serve as an IRT-based content validity index representing SME-evaluated features such as item relevance. The θ^GRM follows a standard normal distribution, allowing for comparisons between items and facilitating the selection of the most appropriate items. For absolute decisions, such as determining if an item is suitable or not (e.g., Aiken’s V > 0.5), model-based expected responses can be computed using predicted probabilities for each judge (Equation 5). Subsequently, CVI and Aiken's V can be calculated using these expectations. With even-category ratings, a CVI or Aiken's V cutoff of 0.5 with expected responses parallels comparing θ^GRM with the average central threshold across SMEs (e.g., b-2 in four-category scales). I-CVI and Aiken's V based on model expectations (CVI_GRM and Aiken’s V_GRM) will also be explored in this article.

To evaluate the utility of the GRM when gathering validity evidence based on the test content, a real-data study was conducted. This study focused on measuring the relevance of a set of personality items for assessing the Conscientiousness (CO) domain within the revised NEO Personality Inventory framework (NEO PI-R; ^{Costa & McCrae, 1992}). Three objectives are outlined as follows:

For the second and third objectives, CVI and Aiken’s V are used as a baseline for comparison. It is hypothesized that, since the IRT model allows for the consideration of each judge's discriminatory capacity and severity, the new proposal will lead to more precise item relevance indices, explaining a higher proportion of the variance of external criteria defined for objectives 2 and 3.