- Part 1: AI foundations
- Part 2: How LLMs learn
- Part 3: Psychometric AI methods & tools
- Part 4: Psychometric AI hybrid analyses
- Part 5: End-to-end AI psychometric modeling
- Part 6: Critical perspectives and future directions
- References
Part 1: AI foundations
Definition and introduction to foundational concepts in AI psychometrics, positioning transformers as a threshold capability marking a milestone in AI psychometrics. Introduces the substitutability assumption as a working framework for explaining AI psychometric utility.
What is AI psychometrics?Substitutability assumption Understanding LLM designsWhat is an encoder architecture?What is a decoder architecture?What are encoder-decoder architectures?Five criteria for choosing a language model in AI psychometricsPart 2: How LLMs learn
This section explores the core learning mechanisms that enable large language models to acquire and demonstrate psychological understanding and capabilities. Worked numerical examples are provided along with code demonstrating LLM pre-training.
How LLMs learn (1 of 4): TokenizationHow LLMs learn (2 of 4): Forward passHow LLMs learn (3 of 4): Back propagationHow LLMs learn (4 of 4): OptimizationBuild an LLM from the ground up (’Seedling’ case-study) Fine-tuning LLMs: Adapters, LoRA, QLoRA, and related methodsAlignment: What it is, why it matters, how to do itEmergent LLM capabilitiesPart 3: Psychometric AI methods & tools
Use of AI to develop the building blocks of assessments, including agents and multi-agent systems in task analysis, test blueprinting, test administration, scoring, and feedback. The section emphasises the need for new methods to meet conventional testing standards.
Testing standards: Reliability and validity XAI: Faithful and Plausible AI measurementMoral foundations as a use caseDo Moral Foundations matter in business?Semantic convergent and discriminant validityPrompt engineering techniquesAI item generation strategiesGenerating items via an APIGenerating items via local LLMs (Ollama)Retrieval Augmented Generation (RAG)Crash course in transformer-era automated scoringScoring (1 of 3): Zero and Few-shot LLMsPart 4: Psychometric AI hybrid analyses
These are two-stage models combing AI and classical psychometrics with frozen embeddings. Methods include semantic item alignment, pseudo factor analysis, network analysis, and convex hulls for item analysis and automated scoring.
Ground truth in AI psychometricsSemantic item alignmentSemantic item alignment resultsPseudo factor analysis Pseudo factor analysis resultsConvex hull analysesSimulation with artificial crowdsLLMs and psychometric biasPart 5: End-to-end AI psychometric modeling
Fully trainable neural networks where all components learn jointly. Unlike Part 4's frozen embeddings, these models add task-specific heads onto transformer architectures. The entire pipeline from encoding to psychometric output is optimized with backpropagation.
Neural end-to-end architecturesScoring (3 of 3): Neural contrastive pairwise regression (NCPR)Part 6: Critical perspectives and future directions
Limitations, ethical and privacy concerns, including geographic/cultural differences in expectations, as well as future directions.
Historical roots of AI assessment debatesPeep into the future: fine-tuned, multimodal, neuro-symbolicReferences
Next section
Last section
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
