- Same item raw embeddings across models are unrelated
- Procrustes rotation aligns embedding vectors
- AI MTMM models requires encoders yielding different information
- References
One seemingly natural extension of pseudo-modeling is pseudo-multi-trait-multi-method CFA models. The idea here would be that we encode scale items using multiple encoders where the encoders are considered different methods. If these model ‘methods’ give unique insights on the scale, the loadings for the items on the trait factors will be higher than the item loadings on the method factors.
Same item raw embeddings across models are unrelated
To check this we can undertake a small experiment. We take a ‘dark side’ personality measure (Guenole, 2015) which has 50 items and generate item embeddings using two models, all-mpnet-base-v2 and bge-base-en-v1.5. The cosine similarity between two embeddings of an example item "Fight fire with fire" is -.076. Other cosine similarities between items are all similarly close to zero.
MTMM models, particularly those estimated as confirmatory factor models (cf. Widaman, 1985), rely on the “right” amount overlap between the same trait measured with different methods. As many people have now shown, within a given AI encoder, representations of psychological constructs are coherent enough to recover useful factor structures. However, here we see between models the information appears to be organized in different ways.
Procrustes rotation aligns embedding vectors
Is it still the same information? We can check this by rotating the 50 item*768 dimension matrix of embeddings from the BGE encoder so that its direction is maximally aligned with the all-mpnet-base-v2 matrix. This is a Procrustes rotation to psychologists and it’s called embedding alignment by AI researchers. After this rotation we check the cosine similarity between the two encodings of “Fight fire with fire” and the value is .93. It is very closely related information being captured across encoders.
This will come as no surprise to representation learning researchers although the size of the jump might surprise psychologists. The cosine similarities across other items post alignment are also very high, ranging from .87 to .97. This too causes problems for MTMM in the Campbell & Fiske (1959) sense, or at least their CFA versions requiring positive definite matrices, because it is nearly the same information but expressed through different coordinate systems. Fitting MTMM CFA modes after aligning the coordinate systems yields a non-positive definite matrix.
AI MTMM models requires encoders yielding different information
These embeddings are too different in raw space and too similar in rotated space for MTMM encoder models. This is a small matrix and the result may partly reflect overfitting, not just common geometry. However, it seem plausible that at least some of this relationship is that the same representation was learned across encoders. If you’re considering fitting MTMM encoder models choose encoders that are sufficiently distinct that you get cross-encoder relations in the Goldilocks zone that is suited to MTMM modeling. This is not fatal for the approach but is a potential failure mode to watch.
References
Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9(1), 1-26.
Next Section
IRT in machine learning: A review of the transformer era
Last section
Scoring (3 of 3): Neural contrastive pairwise regression (NCPR)
Return home
