Pseudo alpha & omega

• A healthy preoccupation
• Possibilities and challenges of embeddings
• Absence of person sample
• Reasons to proceed
• Substituting correlations with similarities
• Pseudo alpha
• Pseudo omega
• Pseudo omega hierarchical
• Further reflections on omega
• Item level diagnostics
• Conclusion
• References

A healthy preoccupation

Psychometrics has a rightful pre-occupation with reliability. Rightful because a requirement for meaningful interpretation of scores is that an instrument’s readings do not fluctuate arbitrarily and un-synchronized with the measurement target. There are three downstream reflections of this epistemic position that permeate what we do in applied practice.

These are that reliability permits estimating measurement precision via the standard error of measurement; that estimates of reliability allow us to correct observed relationships for measurement error; and that the reliability of a measurement method gives an upper bound on the validity coefficients that can be observed using the method.

Possibilities and challenges of embeddings

The psychometrics of AI systems does not necessarily begin with a sample of respondents providing observed item responses. Analyses may focus on internal representations or model behaviour. These set-ups have different implications for the way that we calculate and apply reliability estimates that remain to this point largely unexplored.

Here we discuss that the conventional interpretation of reliability as the proportion of observed-score variance attributable to the true-score rather than measurement error does not apply to representations, despite that the same calculations can be performed on cosine and other similarity matrices in the same way they can be applied to correlation matrices.

Familiar calculations on a cosine similarity matrix should therefore not be considered reliability in the conventional sense. These new quantities might instead be considered indices of semantic cohesion. While the same calculations are being performed, they are performed on different inputs and producing quantities with different interpretations.

The pseudo methods proposed in this section apply to learned representations. Conventional coefficients can be calculated on model behaviour because item responses are available. This section concludes that calculations analogous to human response reliability but performed similarity matrices are important in the context of learned representations. We show worked examples calculating pseudo measures of ‘reliability’ from embeddings.

Absence of person sample

When learned representations are analyzed, precision estimates using the standard error of measurement disappear as a motivation for reliability. In contrast, when model generated responses are analyzed, conventional coefficients can be calculated from response matrices reflecting model consistency rather than human consistency. Repeat administrations also permit conventional test–retest analyses, which is not the case with deterministic representations.

Classical attenuation correction is unnecessary in the learned representation case because there are no observed scores. Whether these quantities provide an upper bound on validity coefficients involving embeddings is unclear. In any case, latent estimates are an output of pseudo factor models. As reliability is frequently higher than validity, one might question the value of learned representation reliability altogether.

Reasons to proceed

An important reason to proceed is pseudo reliability calculations applied to learned representations might tell us about the coherence of these representations or yield pre-knowledge of the human response reliability of a scale. This may include the eventual human standard error of measurement and upper bound on observed human correlations.

One way that has been proposed to get scale reliability from embeddings is building prediction models. This appeals at first encounter because reliability has strong foundations and cultural practices that carry the advantages of being widely accepted. Saying we are predicting reliability from embeddings rather than calculating the quantity itself is less jarring. Yet there is another way.

Substituting correlations with similarities

Following the proposal to substitute the respondent correlation matrix with an item embedding similarity matrix and perform factor analysis (Guenole et al., 2024), we can calculate quantities mathematically analogous to reliability coefficients such as Cronbach’s alpha. By fitting the pseudo factor models we can calculate quantities analogous to McDonald’s omega.

These values may be informative about their human response counterparts and potentially even useful in their own right. We now explore each in turn providing worked examples with data and code. We show how these new values correspond to actual reliabilities and discuss how interpretation differs between embeddings and human responses.

A cosine similarity matrix is symmetric, positive semidefinite, and bounded [-1,1], which mechanically at least, permits the application the reliability formulas explored here. At all times we note that cosine similarities are not derived from respondent sampling so conventional interpretations and benchmarks may not apply.

We emphasise here as we proceed that despite that we can apply existing formulas to cosine matrices, embeddings still reflect co-occurrence patterns in training data rather than psychological response processes. The strength of correspondence between pseudo and human reliability coefficients is going to be a question for empirical research.

Pseudo alpha

The discussion here applies to representation based analyses. The for alpha transfers directly when self-similarities that appear on the diagonal of the similary matrix play the role of item variances and self-other similarities on the off diagonals play the role of covariances. It can be straight forwardly applied to the cosine similarity matrix using the regular formula:

α = (k / (k−1)) × (1 − Σδᵢ / Σsᵢⱼ)

Here K is the number of items, Σδᵢ is the sum of diagonal similarities and Σsᵢⱼ is the sum of all elements in the cosine similarity matrix. The first part of the formula is a correction factor that adjusts for the number of items. The second part of the formula is one minus the ratio of the diagonal similarities to the total similarity in the cosine similarity matrix.

When the number of items is small, the correction has its largest effect on the ratio component of the formula. High values indicate greater similarity among items in the embedding space. If embedding and human quantities using this calculation were related, it may serve as a fast zero-cost screen during scale development before human responses are collected.

Pseudo and real alpha for each of the six scales of the G50 are now presented. There is no direct analogue of the item-total correlation for pseudo alpha because there are no person scores. We propose the mean of an item's cosine similarity with all other items in its scale as an analog, and present this alongside the corrected item-total correlation from the response data.

We also present alpha-if-item-deleted computed by removing each item in turn from the relevant sub-matrix or response matrix. We note that the Pearson and Spearman correlations were r = .81 and ρ = .89 respectively.Sample size is the number of scales or any rank-order comparison, so conclusions about the correspondence must be treated with caution.

Pseudo omega

It is possible to estimate model-based reliability once a factor model’s parameters are estimated. Model based reliability is preferred in response based psychometrics because it relaxes the often-unrealistic assumption of tau equivalence. Analogously, it is possible to compute pseudo omega from the estimated pseudo parameters. The formula for omega is as follows.

ω = (Σλᵢ)² / [(Σλᵢ)² + Σδᵢ] Where λᵢ are factor loadings and δᵢ are uniqueness variances.

Pseudo omega was estimated by fitting a six-factor model to the full G50-item cosine similarity matrix using ML estimation followed by oblique Procrustes rotation using Lavaan. Factor-to-scale correspondence was established using the Dominant Average Absolute Loading (DAAL) criterion. The same procedure was applied to the human response data. Fit for human responses was: χ²(940) = 1954.84, p < .01, CFI = .90, TLI = .86, RMSEA = .044, SRMR = .032. Pseudo model SRMR was .046. Pseudo omegas were lower than response-based counterparts. Pearson and Spearman correlations between pseudo and real omega across the six scales were r = .71 and ρ = .54 respectively noting low N precludes strong conclusions.

Pseudo omega hierarchical

Omega hierarchical can be calculated if a bifactor model is fitted and only the sum of squared loadings on the general factor is included in the numerator. In the pseudo case the quantity might instead be interpreted as the proportion of variance in the embedding similarity structure attributable to the dominant factor. The formula in both instances is: ωh = (Σλg)² / [(Σλg)² + Σλs² + Σδᵢ] Where λg are general and λs are specific loadings. We estimated human ω_h by fitting a bifactor model to the covariance matrix using ML with one general and six specific orthogonal factors. The same procedure was applied to cosine similarites. Human fit was: χ²(896) = 1677.80, p < .01, CFI = .92, TLI = .89, RMSEA = .040, SRMR = .029. Pseudo model SRMR was .042. Omega hierarchical was computed for the general factor. Pseudo ω_h was .94 and human ω_h was .88. Here pseudo ω_h exceeded its human counterpart, suggesting that a single dominant semantic dimension accounts for a particularly large proportion of composite variance in the embedding space.

Further reflections on omega

In single-time-point factor analysis, uniqueness reflects item-specific variance and random error. In human responses these can be disentangled by adding a measurement point. Correlating residuals between the same item at different times models item specific variance that is not residual error.

If we were to run such a model with cosine similarities based on perfectly reproducible sentence encoder embeddings, the longitudinal model would be unidentified because the repeated measurements would be identical, assuming perfect reproducibility and adequate model fit.

Uniqueness in the single-time case may interpreted primarily as item-specific variance rather than random measurement error. Otherwise, uniqueness will also reflect variance from imperfect reproducibility and model misfit. For generative models, this assumes a fixed seed, temperature, and prompt but in that case alpha and omega can be estimated from the resulting item response correlation matrix.

Results so far allow only preliminary interpretation because the effective sample size for any comparison between is equal to the number of scales. Pooling scales across multiple inventories could increase this sample size, but that’s of limited practical value to the applied scale developer, who cannot draw on data from unrelated measures to evaluate their own instrument.

Item level diagnostics

Item-level diagnostics such as alpha-if-item-deleted, item-total correlations, and their pseudo analogues, are presented in the GitHub files but are not discussed in detail here. They do not always show consistent item-level correspondences between pseudo and human response indices and their interpretation under the embedding framework is at this point unclear.

Conclusion

Analogous operations to those performed on a correlation matrix to obtain reliability can be performed on a cosine similarity matrix amongst embeddings. These are operations produce different quantities and are not reliability in the conventional sense, despite the calculations being identical.

Nonetheless, these quantities may prove useful in several respects. They may serve as indices of the internal coherence of scales underlying AI systems, perhaps indicating upper bounds within the system of learned representations. They may afford pre-knowledge of the reliability of corresponding human response-based measures. Alternatively, pseudo alpha on model responses may serve as an indicator of model behavioural consistency.

References

McDonald, R. P. (2013). Test theory: A unified treatment. psychology press.

Raykov, T., & Marcoulides, G. A. (2016). Scale reliability evaluation under multiple assumption violations. Structural equation modeling: a multidisciplinary journal, 23(2), 302-313.

Next section

Scoring (2 of 3): frozen embeddings, downstream predictions

Last section

Pseudo-configural, metric and scalar invariance

Return home

Psychometrics.ai

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).