In this section we show an example of frozen hybrid scoring from the taxonomy discussed in crash course in transformer scoring for unstructured data. We use a sentence encoder to get embeddings of candidate constructed responses and build models to predict human scores of the responses. We will again use the ASAP-SAS data and specifically the question 10 subset. Before we begin, there are several points that are worth discussing.
These models are frozen hybrids in the context of the psychometrics.ai crash course taxonomy because unlike the zero and few shot models, these models fuse the transformer with a prediction head. Unlike the neural end-to-end examples like neural contrastive pairwise regression, no encoder weights are updated via back propagation.
The encoder here is used only as a static feature extractor. Supervised learning occurs only in the model head fitted on top of the frozen embeddings. If the encoder was trained on similar data to the encoding task; this can work well. If the training and task differ, fine-tuning may be needed (Peters et al., 2019).
Frozen transformer embeddings are one way to score constructed responses. The encoder provides fixed text representations, and a separate head models the relationship between those representations and human scores. There is a history of methods using hand crafted features that are not always out-performed by embedding models. Embeddings can be used in stacked models where they augment these hand-crafted features. Hand-crafted feature models are sometimes preferred because of their interpretability (see Haller et al., 2022 for a review).
This section is not intended as a search for the state of the art (SOTA) scoring model in this class for ASAP-SAS Q10. Instead, it is a worked example of a frozen-embedding scoring pipeline: a pre-trained sentence encoder is used as a fixed feature extractor, and supervised model heads are trained on top of those fixed representations. The aim is to show what this relatively cheap and reproducible baseline can achieve before moving to rubric engineering, feature engineering, retrieval, ensembles, or fine-tuning. You can download the notebook containing the analyses for this.
Seven supervised scoring heads
We will compare seven model heads practitioners often try with frozen embeddings: ordinal logistic regression to capitalize on ranked structure; nominal logistic regression which is order agnostic; linear and non-linear support vector machines to maximises margins (gap) between classes; a multilayer perceptron as a simple neural scoring head models nonlinear relationships; and random forests and gradient boosting as tree-based ensemble methods.. We will try all with and without principal component reduction.
Embeddings are often linearly probe-able, meaning they reveal linear relationships between their combined dimensions and a criterion. This is a good reason to expect the linear models to perform well. In contrast, we might expect tree based methods such as random forests to perform less well since they split on individual dimensions, and approximating a truly linear, distributed signal requires many such splits. Gradient boosting is also tree-based, but by correcting residual errors it can be more competitive than random forests.
Pre-processing
It is not clear that PCA before supervised scoring heads will improve performance. PCA is un-supervised and so risks discarding criterion information. Sentence embedding spaces are often anisotropic, and score-relevant signal may be distributed across many dimensions, including lower-variance dimensions that PCA could lose unless dominant components are discarded.
Norming removes magnitude differences but does not alter directions, so it cannot correct anisotropy and many embeddings are already normed in any case. Whitening goes further than PCA and norming by de-correlating the dimensions and rescaling to unit variance. We do not use whitening here because early tests showed that standardization, which is a milder variance-equalizing step hurt performance.
Method and data
We evaluated the models on Question 10 from the Automated Student Assessment Prize Short Answer Scoring dataset (ASAP-SAS), the same short-answer item used in the previous prompt-based scoring example. This contains 1,640 scored student responses, with scores ranging from 0 to 2. For each outer fold, the data were split into 80% outer-training data and 20% outer-test data. The outer-training portion was then split again, with 75% used to fit candidate models and 25% used to select the hyper-parameter value. This gives an effective 60/20/20 train-validation-test structure within each fold. After hyper-parameter selection, the selected model was refit on the full 80% outer-training data and evaluated once on the held-out 20% outer-test data.
Sentence encoder
We used all-MiniLM-L6-v2, a pre-trained sentence-transformer model, to generate frozen 384-dimensional embeddings for each response (Reimers & Gurevych, 2019; Wang et al., 2020). This is a well-known, light and general-purpose encoder that encodes the dataset locally in seconds. Using it in a non-fine-tuned way keeps the encoder constant across models so differences are due to the choice of classifier. All-mpnet-base-v2, a larger encoder, performed no better in terms of accuracy compared to the results we report here (Reimers & Gurevych, 2019; Song et al., 2020).
Classifier Implementation
Classifiers were implemented in Python using scikit-learn (Pedregosa et al., 2011) for nominal logistic regression, linear SVM, RBF SVM, multilayer perceptron, random forest, PCA preprocessing, and cross-validation utilities. Ordinal logistic regression used the mord package (Pedregosa et al., 2017), and gradient boosting used xgboost (Chen & Guestrin, 2016).
Training, validation and testing sub-sampling
To ensure this train/validation/test procedure produced a stable, reliable estimate of performance rather than one dependent on a lucky or unlucky split, we repeated it using 5-fold cross-validation, so that every response served as test data exactly once per repetition, and we repeated this entire 5-fold procedure 10 times with different random partitions of the data, yielding 50 total train-tune-test evaluations per model.
QWK performance metric
Quadratic Weighted Kappa (QWK) is our outcome measure for the comparisons (Cohen, 1960, 1968). This extends Cohen's Kappa, a statistic correcting for the level of agreement expected by chance. QWK adds a penalty term proportional to the squared distance between the predicted and true score categories. A model that confuses a 0 for a 2 is penalized more heavily than one that confuses a 0 for a 1 whereas raw accuracy an un-weighted treats every misclassification the same. Quadratic weighted kappa was computed with sklearn.
Nadeau and Bengio (2003) corrected comparisons
To compare models accounting for the fact that our 50 evaluations were not independent (training sets overlap across the 50 re-samples) we used the corrected resampled t-test proposed by Nadeau and Bengio (2003) rather than a standard paired t-test. The Nadeau-Bengio correction addresses this by inflating the variance term used to compute the t-statistic with an additional factor reflecting the proportion of data held out for testing relative to training.
Results
The 2D visualization of the embeddings is a t-SNE plot that shows the 1,640 Q10 responses projected into two dimensions, coloured by score. The motivation is to check whether the embeddings contain signal related to scores before trusting a classifier built on them. While distinct clusters form, colours (i.e., scores) mix substantially within cluster.
There is likely enough signal for a classifier to do better than chance but perhaps not enough for highly accurate scoring. Keep in mind that t-SNE is a nonlinear projection and used for intuition here only. It represents local relationships well but can distort global distances and the clusters that emerge can change sensitive to perplexity.
Across the seven classifiers and two preprocessing conditions (14 combinations in total), each tuned within the outer training fold using a stratified validation split and evaluated across 50 repeated train-validation-test fold-evaluations, Nominal Logistic Regression on raw embeddings produced the highest mean Quadratic Weighted Kappa (QWK = 0.711).
In general, simplicity was rewarded: linear-additive models (both logistic regression variants and the linear-kernel SVM) matched or outperformed models capable of representing nonlinearity, including the RBF-kernel SVM, Random Forest, Gradient Boosting, and the Multilayer Perceptron which likely needed more data and better tuning.
The top-performing model is Nominal LR on raw embeddings, with several linear models (Linear SVM, SVM, Ordinal LR) showing competitive results. Dimensionality reduction via PCA(taking first 50 components) provides little to no benefit for most classifiers and hurts MLP performance. Random Forest models are the only ones significantly worse than the best configuration according to the Nadeau-Bengio corrected pairwise p-values.
This heat map presents mean QWK scores by model and preprocessing strategy. Nominal LR achieves the highest performance on both raw and PCA-reduced embeddings. PCA(50) yields minimal benefit for most models and substantially degrades MLP performance. Colour intensity reflects QWK value.
The final ranking confirms Nominal LR on raw embeddings achieves the highest performance with a mean QWK of 0.7114. Several linear models, including Linear SVM and SVM on raw features, follow closely with negligible differences . Most configurations are statistically comparable to the best model, but Random Forest variants are significantly worse. PCA(50) preprocessing offers little to no improvement and sometimes worsens performance.
Interpretations
The result falls below human-human agreement (QWK = 0.884), so the accuracy is lower than the consistency of two trained human raters. Zero-shot and few-shot prompting of GPT-4 on a subsample of this essay set achieved QWK values of 0.733 and 0.750 respectively in an earlier section. This level might be acceptable for hybrid deployment with human assistance but accuracy can likely be refined with work. Higher performance has been achieved using non-transformer feature engineering with semantic features and models stacking.
There is little separating the winning methods, suggesting once you have decent embeddings the classifier choice has less effect. The tree based methods did not perform as well as linear models, indicating they struggle to approximate what the other models show are linear relationships. When comparing to reported QWK in other papers check if their train, validate, test split was not lucky. Here we average over 50 repetitions of design.
Taken together, these comparisons show that frozen embeddings with simple heads capture a reliable association between text and score at minimal computational cost and with no task-specific engineering. While approaches that add feature engineering, fine-tuning, or LLM prompting achieve higher performance, the frozen-embedding results are a cheap baseline on what is achievable.
References
Chen, T., & Guestrin, C. (2016, August). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining (pp. 785-794).
Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4), 213.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46.
Haller, S., Aldea, A., Seifert, C., & Strisciuglio, N. (2022). Survey on automated short answer grading with deep learning: From word embeddings to transformers. arXiv. https://arxiv.org/abs/2204.03503
Nadeau C. & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3):239–281, 2003.
Pedregosa, F., Bach, F., & Gramfort, A. (2017). On the consistency of ordinal regression methods. Journal of Machine Learning Research, 18(55), 1-35.
Pedregosa, Fabian, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel et al. "Scikit-learn: Machine learning in Python." the Journal of machine Learning research 12 (2011): 2825-2830.
Peters, M. E., Ruder, S., & Smith, N. A. (2019). To tune or not to tune? Adapting pretrained representations to diverse tasks. Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), 7–14. https://aclanthology.org/W19-4302/
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP). https://arxiv.org/abs/1908.10084
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020). Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33, 5776-5788.
Next page
Last page
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
