Pseudo factor analysis

Until now, pseudo-factor analysis has been applied to pre-existing items and scales, but we haven’t presented a ground up application of scale development. Here we’ll do that, showing pseudo-discrimination and pseudo factor analysis work nicely as a data-less method for obtaining pre-knowledge of item discrimination with A.I generated items.

What is pseudo factor analysis?

Pseudo factor analysis is a method that allows us to do this. At the heart of pseudo factor analysis is the “substitutability assumption”, or the idea that the embedding vector for an item statement can stand in for an empirical response vector. We defer discussion of the reasonableness of this claim in different circumstances to other sections of this book.

For now, we simply say that once the item embedding vectors have been generated, a cosine similarity matrix can be formed, and the cosine similarity matrix can be analyzed with factor analysis in essentially the same way that a correlation matrix of from real item responses is analyzed. Pseudo factor analysis variation was proposed by Guenole et al. (2025). A related principal component variation was reported by Milano et al. (2025). A network based analysis of cosine similarity matrices was reported by Russell-Lasalandra et al. (2024).

Step 1. Generate item embeddings

Once your item pool is defined and language models are selected, the next step is to transform each item into a numerical representation, or embedding. These embeddings capture the semantic meaning of each item based on the model’s understanding of language. There are three ways to approach this step.

The Atomic method involves generating embeddings for each item individually. The Atomic Reversed method builds on this by multiplying embeddings by their theoretical item sign before aggregation. Lastly, the Macro method involves concatenating all items within a facet and generating a single embedding for that combined text.

Step 2. Construct similarity matrices

The next step is to evaluate how semantically similar each item (or facet) is to every other. This is done by creating a similarity matrix, where each cell represents the cosine similarity between a pair of embeddings. Cosine similarity ranges from -1 to 1 and reflects how closely aligned two vectors are, with higher values indicate greater semantic similarity.

If multiple language models were used, produce a similarity matrix for each and then aggregate them by averaging into a single matrix. The resulting matrix, which captures the semantic structure of the item pool, will serve as a substitute for the empirical correlation matrix traditionally used in factor analysis. This matrix is now ready for pseudo-factor analysis.

Step 3. Conduct exploratory factor analysis

Choose a suitable extraction method like Maximum Likelihood to identify factors, and specify the number of factors to extract based on theory. Since psychological constructs are often correlated, apply an oblique rotation method, potentially with target rotation, to make interpretation easier. The analysis will produce a factor loading matrix.

Step 4. Interpret fit and factor loadings

Pseudo factor analysis does not yield a sample size for chi-square tests and related fit indexes. Arbitrary assignment of sample sizes should therefore be avoided, as chi-square and related indexes are highly dependent on sample size for accuracy. Model evaluation is better guided by inspecting unstandardised residuals, their plots, and related statistics to identify areas of good and poor fit.

Factor loadings indicate how strongly each item or facet is associated with each extracted factor. To systematically assign items to factors, use the Dominant Average Absolute Loading (DAAL) approach. This involves calculating the average of the absolute loadings for all facets theoretically expected to belong to each factor and determining which factor has the highest average.

If an item (or facet) clearly loads onto one factor, it can be assigned there. Some items may load similarly on multiple factors or fail to load strongly on any. These are considered unassigned or merged factors. These patterns highlight potential issues with item clarity or construct coverage that may need to be addressed before proceeding.

Step 5. Refine item pools if necessary

Look for items or facets that cross-load onto multiple factors, fail to load strongly on any factor, or are unassigned according to DAAL in the factor solution. Decide whether to revise the wording, reassign items to other constructs, or remove them altogether. The goal is to get a clean structure before collecting empirical data.

Step 6. Empirical validation

This step bridges the gap between the AI-generated semantic structure and real-world psychological measurement. Start by obtaining empirical factor loading matrices from published studies or previously collected data using human responses. Then, compare these to the pseudo-factor loadings using quantitative metrics.

Two key metrics are Tucker’s congruence coefficient, which assesses the similarity of factor patterns (values above .85 indicate fair similarity and values above .95 indicate strong alignment), and correlation coefficients, which reveal how strongly corresponding factors relate across empirical and pseudo models.

Code to run pseudo factor analysis and an empirical demonstration follows in the next section. For an online app that generates items and then checks their factor structure using pseudo factor analysis, check out the Psicometrista app from Damiano D’Urso.

Moral foundations sunburst plot

Click the interactive link to explore the sunburst plot of items and their relative contribution to each of the moral foundations factors discussed in the earlier section of this book. The size of the sectors in the plot reflects size of each item’s relative contribution to its factor. You can interact with the plot to see the content of each of the items themselves for each of the six scales.

The sunburst plot is a useful vizualization of the item-to-factor relationship, it misses the critical issue of the level of item contributions to other factors, or cross loadings. We hope that these are small for better measurement, but we need to check. To do this we examine the pattern coefficients from the oblique rotated factor solution itself.

Factor loading table

These are important to show that items reflect the intended construct and so far as possible no other construct. We see that this is the case for all items in the table, the dominant loading for each item is on its target factor and that loading is greater than its loading of any other factor and greater than the average all its loadings on all other factors.

Overall, this table shows that we have a clean factor structure measuring moral foundations scales that we can use to move to empirical item trialing. As with conventional scale development, we still expect some items to be removed based on empirical analysis. We will generate back up items to trial with fine tuned item generation, which we turn to next.

Next section

References

Guenole, N., D’Urso, E. D., Samo, A., Sun, T., & Haslbeck, J. M. (2025). Enhancing scale development: Pseudo factor analysis of language embedding similarity matrices.

Milano, N., Luongo, M., Ponticorvo, M., & Marocco, D. (2025). Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality tests. Current Research in Behavioral Sciences, 8, 100168. https://www.sciencedirect.com/science/article/pii/S2666518225000014

Russell-Lasalandra, L. L., Christensen, A. P., & Golino, H. (2024). Generative psychometrics via AI-GENIE: Automatic item generation and validation via network-integrated evaluation. PsyArXiv Preprints.

Next page

Pseudo-factor analysis with mean structures

Last page

Semantic item alignment

Return home

Psychometrics.ai

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).