What is AI psychometrics?

What is AI psychometrics?

It’s about transformers

Psychometrics has always had strong relationships with quantitative fields like statistics and the computer science subfields of machine learning (ML) and artificial intelligence (AI) (e.g., von Davier, 2017). While ML and early AI have been used in assessment for some time, AI made large strides in around 2017 due to the arrival of transformer models. They were introduced in a landmark paper by Vaswani et al., (2017).

The use transformer models became widespread in the early to mid 2020s but there were notable examples of these methods applied soon after their emergence in educational testing settings (e.g., von Davier, 2019; Laverghetta et al., 2021; Ormerod et al. 2021). This pioneering work with transformers for psychometrics demonstrated their utility for a wide range of psychological measurement tasks.

Transformers as a threshold capability

We will use the integration of transformers into psychometric workflows as the key marker distinguishing AI psychometrics from earlier computational psychometric methods. Transformers can efficiently process text, audio, and video, allowing versatile applications across the assessment lifecycle. Current applications can be classed as representational or natural language understanding (NLU) focused, or as using generative AI models for common psychometric tasks.

AI psychometrics definition

AI psychometrics applies transformers, including Large Language Models, across the entire psychological assessment process, including mapping skills and roles, designing and scoring tests, translating assessments, generating feedback, making predictions, and analyzing outcomes, improving the precision and impact of psychological measurement.

💡

A Transformer is a neural network architecture that uses tokenization, embeddings, and self-attention to create context-aware representations of sequence-based data and to generate new text. Large Language Models (LLMs) are transformers trained on natural language data.

Other important forms of AI

We acknowledge that other forms of AI, such as classical machine learning models (e.g. support vector machines, decision trees, and their variations like random forests and gradient boosting) have played important roles in psychometrics and continue to do so today. Research into these areas and their applied use is an active area within the psychometric community. In practice, transformers are often integrated with these powerful non-transformer methods. While this is a book about transformers for psychometrics, we include a section on how transformers are being integrated with these alternative forms of AI in a section on hybrid AI models.

Uncertainty with opportunities

Today, transformers are used alongside conventional methods in the design and delivery of assessments. However, the extent of transformer use relative to pre-transformer approaches is unclear, their effectiveness is a new area of investigation. There is concern about bias in most applications of large language models, there are questions about best practices (e.g., in translation) and there is regulatory uncertainty.

Notwithstanding caveats, which are common to many new technologies, transformers are proving effective tools in many areas of psychometrics. In this book we will explore new applications in of AI, offer reproducible examples of AI psychometrics with code, and try to offer a critical view of the strengths and limitations of what, at the end of the day, is an exciting new development for psychological measurement.

From conventional to AI assessment

Here are some examples how these methods are being used in the assessment context.

  • Automated item generation (AIG) using transformers (Hernandez & Die, 2022; Hommel et al., 2023)
  • Item construct validity checks using sentence encoders (Guenole et al., 2024)
  • Pre-knowledge of item parameters using embedding analyses (Guenole et al., 2025; Laverghetta et al., 2021; Russell-Lasalandra et al., 2024)
  • AI scoring of constructed response format assessments (Casabianca et al., 2025)
  • Use of AI by job candidates to cheat (Canagasuriam et al., 2025)
  • Automated examination of convergent and divergent validity (Wulff & Matta, 2025)
  • Automated candidate feedback (Dai et al., 2024)
  • Assessment language translations (Hickman et al., 2022)
  • Automated interview scoring (Jung et al., 2024)

Two-way information exchange

This list of applications is extensive, and it may seem like the flow of techniques is a one way street from computer science to psychometrics. The focus of these pages is indeed on the application of computer science methods in psychometrics, with an emphasis on industrial psychology contexts. But there are also examples of where ideas from psychometrics are being applied to improve the function of transformer models. For example, here are two examples of this happening.

  • Bias detection in language models (Shrestha et al., 2025)
  • AI character training (Financial Times, 2025)

Next section

Reasons for AI psychometric progress

Last section

Table of contents

Return home

Psychometrics.ai

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).