Until now, pseudo-factor analysis has been applied to pre-existing items and scales, but we haven’t presented a ground up application of scale development. Here we’ll do that, showing pseudo-discrimination and pseudo factor analysis work nicely as a data-less method for obtaining pre-knowledge of item discrimination with A.I generated items.
What is pseudo factor analysis?
Pseudo factor analysis is a method that allows us to do this. At the heart of pseudo factor analysis is the “substitutability assumption”, or the idea that the embedding vector for an item statement can stand in for an empirical response vector. We discuss the reasonableness of this claim in different circumstances at the end of this section.
For now, we simply say that once the item embedding vectors have been generated, a cosine similarity matrix can be formed, and the cosine similarity matrix can be analyzed with factor analysis in essentially the same way that a correlation matrix of from real item responses is analyzed.
Step 1. Generate item embeddings
Once your item pool is defined and language models are selected, the next step is to transform each item into a numerical representation, or embedding. These embeddings capture the semantic meaning of each item based on the model’s understanding of language. There are three ways to approach this step.
The Atomic method involves generating embeddings for each item individually. The Atomic Reversed method builds on this by multiplying embeddings by their theoretical item sign before aggregation. Lastly, the Macro method involves concatenating all items within a facet and generating a single embedding for that combined text.
Step 2. Construct similarity matrices
The next step is to evaluate how semantically similar each item (or facet) is to every other. This is done by creating a similarity matrix, where each cell represents the cosine similarity between a pair of embeddings. Cosine similarity ranges from -1 to 1 and reflects how closely aligned two vectors are, with higher values indicate greater semantic similarity.
If multiple language models were used, produce a similarity matrix for each and then aggregate them by averaging into a single matrix. The resulting matrix, which captures the semantic structure of the item pool, will serve as a substitute for the empirical correlation matrix traditionally used in factor analysis. This matrix is now ready for pseudo-factor analysis.
Step 3. Conduct exploratory factor analysis
Choose a suitable extraction method like Maximum Likelihood to identify factors, and specify the number of factors to extract based on theory. Since psychological constructs are often correlated, apply an oblique rotation method, potentially with target rotation, to make interpretation easier. The analysis will produce a factor loading matrix.
Step 4. Interpret factor loadings
Factor loadings indicate how strongly each item or facet is associated with each extracted factor. To systematically assign items to factors, use the Dominant Average Absolute Loading (DAAL) approach. This involves calculating the average of the absolute loadings for all facets theoretically expected to belong to each factor and determining which factor has the highest average.
If an item (or facet) clearly loads onto one factor, it can be assigned there. Some items may load similarly on multiple factors or fail to load strongly on any. These are considered unassigned or merged factors. These patterns highlight potential issues with item clarity or construct coverage that may need to be addressed before proceeding.
Step 5. Refine item pools if necessary
Look for items or facets that cross-load onto multiple factors, fail to load strongly on any factor, or are unassigned according to DAAL in the factor solution. Decide whether to revise the wording, reassign items to other constructs, or remove them altogether. The goal is to get a clean structure before collecting empirical data.
Step 6. Empirical validation
This step bridges the gap between the AI-generated semantic structure and real-world psychological measurement. Start by obtaining empirical factor loading matrices from published studies or previously collected data using human responses. Then, compare these to the pseudo-factor loadings using quantitative metrics.
Two key metrics are Tucker’s congruence coefficient, which assesses the similarity of factor patterns (values above .85 indicate fair similarity and values above .95 indicate strong alignment), and correlation coefficients, which reveal how strongly corresponding factors relate across empirical and pseudo models.
Next page
Pseudo factor analysis results
Last page
Semantic item alignment results
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).