Search

Book purpose

Table of contents

Note of thanks

Endorsements

References

License and referencing

Update tracking

Table of contents

Table of contents

  • Part 1: AI foundations
  • Part 2: How LLMs learn
  • Part 3: Psychometric AI methods & tools
  • Part 4: Psychometric AI hybrid analyses
  • Part 5: End-to-end AI psychometric modeling
  • Part 6: Critical perspectives and future directions
  • References

Part 1: AI foundations

Definition and introduction to foundational concepts in AI psychometrics, positioning transformers as a threshold capability marking a milestone in AI psychometrics. Introduces the substitutability assumption as a working framework for explaining AI psychometric utility.

What is AI psychometrics?Substitutability assumption Understanding LLM designsWhat is an encoder architecture?What is a decoder architecture?What are encoder-decoder architectures?

Part 2: How LLMs learn

This section explores the core learning mechanisms that enable large language models to acquire and demonstrate psychological understanding and capabilities. Worked numerical examples are provided along with code demonstrating LLM pre-training.

How LLMs learn (1 of 4): tokenizationHow LLMs learn (2 of 4): forward passHow LLMs learn (3 of 4): back propagationHow LLMs learn (4 of 4): optimizationSeedling: LLM training walkthrough Fine-tuning LLMs: Adapters, LoRA, QLoRA, and related methodsAlignment: what it is, why it matters, how to do itEmergent LLM capabilities

Part 3: Psychometric AI methods & tools

Use of AI to develop the building blocks of assessments, including agents and multi-agent systems in task analysis, test blueprinting, test administration, scoring, and feedback. The section emphasises the need for new methods to meet conventional testing standards.

Testing standards Faithful and Plausible AI MeasurementMoral foundations as a use caseDo Moral Foundations matter in business?Semantic convergent and discriminant validityAI item generation strategiesGenerating items via an APIRetrieval Augmented Generation (RAG)

Part 4: Psychometric AI hybrid analyses

These are two-stage models combing AI and classical psychometrics with frozen embeddings. Methods include semantic item alignment, pseudo factor analysis, network analysis, and convex hulls for item analysis and automated scoring.

Semantic item alignmentSemantic item alignment resultsPseudo factor analysis Pseudo factor analysis resultsConvex hull analysesExploring embedding compositionsSimulation with artificial crowdsLLMs and psychometric bias

Part 5: End-to-end AI psychometric modeling

Fully trainable neural networks where all components learn jointly. Unlike Part 4's frozen embeddings, these models add task-specific heads onto transformer architectures. The entire pipeline from encoding to psychometric output is optimized with backpropagation.

Part 6: Critical perspectives and future directions

Limitations, ethical and privacy concerns, including geographic/cultural differences in expectations, as well as future directions.

Historical roots of AI assessment debatesPeep into the future: fine-tuned, multimodal, neuro-symbolic

References

References

Next section

What is AI psychometrics?

Last section

Book purpose

Return home

Psychometrics.ai

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).

image