- Autoregressive multi-trait scoring (ArTS)
- Architecture choice
- Data preparation
- Predictors and target variable
- Fine-tuning
- Score extraction
- Results
- Why not just use an LMM for this for scoring?
- References
Autoregressive multi-trait scoring (ArTS)
Transformer scoring approaches we have covered so far in this book have relied on encoders or decoders and required creating separate models for different questions. Recent work has instead specified scoring with transformers as an autoregressive text generation task using encoders-decoder models and created single scoring models that apply across different questions.
Here we examine and rebuild the autoregressive scoring pipeline described by Do et al. (2024). They showed that a single scoring model can be trained to score responses to different prompts where essays are scored along multiple dimensions. To test the write up clarity, we build a ground up implementation without looking at their code. We consider this an implementation rather than replication since we make several different design choices. Our code is here.
These authors refer to “trait” scoring, but it is really a rubric with multiple analytical dimensions. While this method emerges from education, there are strong parallels with constructed response situational judgement tests, video interviews and assessment center simulations where candidate responses are scored against a set of competencies. This model can score multiple traits for multiple questions in one model and uses an encoder-decoder, which we will fine-tune.
Architecture choice
The first choice is deciding which architecture to adopt. Encoder-decoders and decoders can be used to generate scores. Encoder/decoders have bidirectional attention for a deeper understanding of the text compared to decoder causal attention. Prefix masking allows bidirectional attention on a decoder prompt but currently has limited implementation options.
In an earlier section on encoder-decoders, we discussed that decoders dominate in modern application of transformers, except in specialized domains such as translation where bidirectional and cross attention help. Scoring free text is another domain where encoder/decoders can outperform and did in the Do et al. that study we implement here.
The Do et al. paper used Google’s T5 encoder-decoder and we will use a model with the same architecture for this demonstration. For showing multi-trait scoring we will use T5-small, rather than T5 base that Do et al used without loss of explanatory cohesion. T5 small has 60m parameters as opposed to T5-Base's 220m parameters.
Data preparation
This ASAP data set needs augmenting for this work because while prompts 7 and 8 contain per-trait scores, prompts 1 through 6 only have a single overall score. For those six prompts augmented the data with trait-level scores from ASAP++ (Mathias & Bhattacharyya, 2018) to create a complete data set covering all 8 prompts with 11 possible trait columns per essay and a gap if a construct doesn't apply to that prompt that is filled with NaN (Not a number).
Predictors and target variable
Unlike our NPCR rebuild, ArTS needs no essay pairing at all. Each essay is scored on its own, from its own text with no reference essays or vote-averaging at inference. Scoring a new essay is one forward pass. The input is the essay's own text, prefixed with its prompt number: "score the essay of the prompt N: [essay text]". Do et al. tested found this improves accuracy.
The target is a single string naming every trait alongside its score, comma-separated, in a fixed order the paper specifies: rarest, most prompt-specific traits first and the overall score last. A trait that doesn't apply to a given prompt gets the value "nan" in this string, the same value the model has to learn to produce, so it is trained to actively predict not a number for these cases rather than having the data analysts omit or fill the score.
Fine-tuning
T5 is pre-trained on general text-to-text tasks so the model has the capability to generate alpha numeric strings. However, the pre-trained model has never seen this exact task. An early test, trained for only a few epochs on a handful of essays produced fragments of the input essay instead of a scored output. Fine-tuning on the full training set is clearly required to teach the model beyond its pre-training capabilities.
We followed the paper's training configuration with batch size and epochs, and five-fold cross-validation. Do et al. (2024) used Taghipour and Ng's (2016) fold split, while we built our own random stratified split. We evaluated validation accuracy once per epoch, computing Quadratic Weighted Kappa (QWK) on the held-out validation set. We did not use early stopping and selected the best-validated checkpoint across all epochs.
Training used the Hugging Face Transformers T5 small model. We used the default AdamW optimiser with batch size of four directly on a single GPU on Google Colab. Inputs were truncated to a maximum of 512 tokens and targets to 64 tokens. Each fold was trained for the full 15 epochs. The checkpoint with the highest validation QWK was then retained as the model for that fold.
Score extraction
We extracted each trait's score from the generated string by name using regex commands since the model doesn't always produce text with traits in the trained order. A value that couldn’t be found might have been omitted, spelled incorrectly, or been non-numeric. In these cases, we recorded the result as a missed extraction which is one of the failure modes of autoregressive score generation.
Results
The implementation achieved a mean trait-wise QWK of .644 using T5-small. This was close to the .651 reported by Do et al. (2024) for the same model size despite using different cross-validation folds. Mean prompt-wise QWK was .684, against .680 in the published study. The mean absolute difference across the eight prompts was .017.
These results indicate that our new implementation closely captures the scoring capabilities described in the ArTS paper despite differences in fold membership and training infrastructure decisions. The largest discrepancy we found in scoring accuracy was for Voice which the original study also identified as difficult (QWK .348 vs .474).
Because we used our own folds rather than the Taghipour & Ng (2016) splits used by Do et al. the numerical comparison is not fully matched. The close agreement (mean absolute difference of .017 across prompts) is encouraging but this was not aimed to be a strict replication, hence our reference to the work as an “implementation”.
An error analysis checked the two failure modes Do et al. (2024) report in their own appendix. Extraction failures were negligible, with only 3 of score predictions across all five folds unable to be parsed from the generated text. Just 21 predictions fell outside a prompt's actual, though this is more than the single case in the original study.
Here we report stability across the five folds. Trait-wise average QWK ranged from 0.618 to 0.669, a standard deviation of 0.019, and prompt wise-average QWK ranged from 0.667 to 0.699, a standard deviation of 0.013. These are both more consistent than the 0.026 and 0.029 Do et al. (2024) report for their own T5 small model.
These QWK values are lower than would usually be needed for automated scoring even though they matched the paper. But this implementation used T5 small, the smallest model Do et al. (2024) tested. For T5 base they report a trait wise mean QWK of .695 and a prompt wise mean QWK of .717 and it was similar for t5 large.
Why not just use an LMM for this for scoring?
There are important practical and psychometric reasons to prefer a model such as the ArTS approach to multi-trait scoring over off-the-shelf LLM prompting, whether that scoring is via API or local inference. ArTS is trained and evaluated against human ratings. General-purpose LLM prompted to score essays have no such constraint. They can sound convincing while still scoring systematically too high or too low or ranking essays inconsistently with human experts. Do et al. (2024) showed that T5 outperformed decoder-only models on this task. The encoder-decoder bidirectional and cross-attention appear better suited to deep understanding of the response text before generating scores. This architecture matches the task. Cost and latency at scale is low. Scoring large volumes of responses with a proprietary LLM is substantially more expensive and runs slower than a single forward pass through a fine-tuned 60–220 million parameter encoder-decoder. Data governance and reproducibility is strong. Many contexts prohibit sending candidate responses to third-party APIs. Even with open-weight models, base weights and prompting behaviour can change across releases; a fine-tuned checkpoint is fully reproducible. Free-form autoregressive generation of score strings need post-processing and can fail. A smaller specialised model like ArTS can be more robust for high-stakes use than unconstrained LLM generation. This t5 model builds in output structure consistency.
References
Do, H., Kim, Y., & Lee, G. G. (2024). Autoregressive score generation for multi-trait essay scoring. arXiv preprint arXiv:2403.08332.
Taghipour, K., & Ng, H. T. (2016, November). A neural approach to automated essay scoring. In Proceedings of the 2016 conference on empirical methods in natural language processing (pp. 1882-1891).
Next page
Negative control: If you can’t simulate, destroy
Previous page
Scoring: Neural pairwise contrastive regression (NPCR)
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
