Objective and audience
This book examines modern artificial intelligence (AI) capabilities through the lens of psychological measurement. It examines AI psychometrics, defined as the application of frontier AI systems to psychological assessment and measurement. The methods discussed are relevant measuring human psychological attributes and to evaluating the capabilities, behaviours, and emergent properties of intelligent AI systems.
This book’s framing includes transformer-based systems, state space models such as Mamba, symbolic AI, and broader machine learning methods within a unified psychometric perspective. Applications of AI psychometrics in talent assessment include competency mapping, skills inference from digital footprints, assessment design, constructed response scoring, translation, feedback generation, and evaluation of intelligent systems.
The book is intended for technically minded applied data scientists, industrial psychologists, researchers, organizational consultants, and HRTech professionals interested in using modern AI architectures to enhance psychological assessment. It offers general insights for business leaders alongside technical content for specialists.
Practical and applied emphasis
The book emphasises ‘learning by doing’ for practical understanding over theory. Material covers the technical knowledge required for a thorough understanding of transformer models. It then moves to material needed for applied psychometrics professionals working with AI. It finishes with presentation of key issues for ethicists and policy makers.
The book shows how to reconstruct operational LLMs from the ground up, how you can train a ‘small’ LLM from scratch, how to fine tune an LLM, how methods such as Reinforcement Learning from Human Feedback (RLHF) work, and how to secure spot pricing from cloud computing vendors.
New transformer based developments in psychological measurement are also discussed in detail, including semantic construct and item validity, pseudo factor analysis, and more experimental methods such as the application of convex hulls to item analysis and free text scoring. I take an intentionally forward-looking view in places, identifying areas where I believe psychologists can make a contribution to both AI itself and AI psychometrics.
Vendor agnosticism
I am a fan of open-source models wherever it makes sense for their transparency and reproducibility and use open source models in many parts of this book. After working in the technology industry for many years, I also recognize that models from the hyperscalers like Microsoft and Amazon Web Services have their attractions.
These providers can offer well supported enterprise grade models that are integrated into corporate technology ecosystems with clear legal protections. Practitioners should also develop versatility in their methodological approach given that it’s very early in the LLM technology cycle. In this book I use a variety of open and closed source models from various LLM and cloud computing vendors.
Writing strategy and AI use
This book is free and continuously updated. My approach is to map the broad terrain of AI and psychometrics in an initial pass, then refine sections, add graphics and illustrations, and carry out mathematical verification.
The book's conceptual framework, structure and core written content were produced by me. LLMs were used for testing whether I understood technical concepts, grammar checking and referencing.
AI coding tools assisted with mathematical formulas and complex matrix calculations, and coding for the twinkle twinkle little star BPE, LLM reconstructions, Seedling case study, and RAG and XAI demonstrations.
AI has been helpful for discrete tasks, but not end-to-end workflows. Human understanding of architectures was required to ensure correct sequencing of models. Results are preliminary and will be verified before the first release.
Next section
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
