- Objective and audience
- Practical and applied emphasis
- Vendor agnosticism
Objective and audience
This book explores AI psychometrics, which we define as the use of transformers, including Large Language Models (LLMs), in psychological assessment. This can include the integration of transformers with symbolic AI (e.g. rule based AI systems) as well as their integration with earlier computational approaches in psychometrics.
Applications of AI Psychometrics include job-competency mapping, skills inference from digital footprints, test design and scoring, translation, feedback generation, and prediction and clustering, all aimed at enhancing precision and impact in psychometrics and people analytics.
The book is intended for applied psychometricians, industrial psychologists, organizational consultants, and HRTech professionals interested in using transformers and Large Language Models to enhance psychological assessment and people analytics. It offers general insights relevant to business leaders alongside technical content for psychometric specialists.
Practical and applied emphasis
The book emphasises ‘learning by doing’ for practical understanding over theory. Material covers the technical knowledge required for a thorough understanding of transformer models. It then moves to material needed for applied psychometrics professionals working with A. It finishes with presentation of key issues for ethicists and policy makers.
The book shows how to reconstruct operational LLMs from the ground up, how you can train a ‘small’ LLM from scratch, how to fine tune an LLM, how methods such as Reinforcement Learning from Human Feedback (RLHF) work, and how to secure spot pricing from cloud computing vendors.
New transformer based developments in psychological measurement are also covered in detail, including semantic item and construct validity, pseudo factor analysis, and more experimental methods such as the application of convex hulls to item analysis and free text scoring. Some sections are well developed and available, others will be added incrementally.
Finally, I take an intentionally speculative tone in places, identifying areas where I believe psychologists can make a future contribution to the AI side of psychological measurement.
Vendor agnosticism
I am a fan of open source models wherever it makes sense for their transparency and reproducibility and use open source models in many parts of this book. After working in the technology industry for many years, I also recognize that models from the hyperscalers like Microsoft and Amazon Web Services have their attractions.
These providers can offer well supported enterprise grade models that are integrated into corporate technology ecosystems with clear legal protections. Practitioners should also develop versatility in their methodological approach given that it’s very early in the LLM technology cycle. In this book I use a variety of open and closed source models from various LLM and cloud computing vendors.
Writing strategy and AI use
This book is live and evolving. My approach is to map the broad terrain of AI and psychometrics in an initial pass, then refine sections, add graphics and illustrations, and carry out mathematical verification.
The book's conceptual framework, structure and core written content were produced by me. LLMs were used for testing whether I understood technical concepts, grammar checking and referencing. AI coding tools assisted with mathematical formulas and complex matrix calculations, the twinkle twinkle little star BPE, and coding for LLM reconstructions and the Seedling case study.
AI has been helpful for discrete tasks, but not end-to-end workflows. Human understanding of architectures was required to ensure correct sequencing of models. Results are preliminary and will be verified before the first release.
Next section
Table of contents
Return home
Psychometrics.ai
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
