- Introduction
- Specifying the architecture
- Data preparation
- Essay pairing
- Target and labels
- Model training
- Model inference
- Results and discussion
- Training
- Validation
- Test
- Note on model deployment
- References
Introduction
In this section we will implement from the ground up an end-to-end neural prediction technique for constructed response scoring called Neural Pairwise Contrastive Regression (NPCR). We focus on the article by Xie et al (2022) and our psychometrics.ai implementation of their model. We coded the version discussed here by reading their description of the method to infer the computational graph, despite a number of model aspects not being discussed in the article. You can get our code with full optimization logs here.
Xie et al. (2022) used NPCR to predict essay scores in the Automated Student Assessment Prize (ASAP) automated essay scoring (AES) challenge. The challenge contains essays scored for writing quality across 8 prompts and was established by the Hewlett Foundation and hosted on Kaggle in 2012. Today it is a classic benchmark dataset in the automated essay scoring psychometrics field.
We will test the NPCR method by trying to recreate their result for essay prompt four, one of the higher accuracy prompts. Implementing the model based on reading the paper led to strong Quadratic Weighted Kappa (QWK) values, despite our implementation differing in several respects on design points where the paper was silent, but a post-re-implementation examination of their repo revealed more details.
Note that the authors’ own code on GitHub differs from their paper description in several respects. We could also not see their optimization history for their strong results. However, our re-implementation here gives us confidence that their method works well. Visit our GitHub and download our implementation, with fold-by-fold, epoch-by-epoch, and batch-by-batch optimization histories that achieve competitive QWK accuracy.
Specifying the architecture
The first step in the architecture is to specify the architecture the data flows through. This starts with a BERT encoder to create embeddings of each of the essays. Next, a Siamese network, two identical multilayer perceptron (MLP) neural networks (nn1, nn2), is specified that transforms the embeddings to a new representation for the next step.
We implemented this as a single shared network called for both essays, since that is the straight forward way to meet the requirement that nn1 and nn2 share weights. Sharing weights between nn1 and nn2, combined with the third MLP having no bias term and an activation function satisfying f(−x) = −f(x) (such as tanh), means that it doesn't matter which essay is first in the pair, it gives the same score difference with just the sign flipped.
Next comes calculation of a difference vector between each pair of embeddings. Finally, a third MLP is added that predicts the outcome, i.e., difference score representing the contrast between the human-assigned scores for the two essays. The difference for the predicted essay is added to the score for the reference essay to give the predicted score.
Data preparation
Before we begin our NPCR model training we need eto prepare the data set. As with our earlier examples on the ASAP-SAS data set, separate models are usually built for different essay prompts. Here we will consider just one essay prompt to illustrate the method, prompt four, which contains 1,780 essays scored 0–3 (1,062 train / 354 validation / 354 test).
With complex models such as this, a helpful start is to understand the data structure: what does the model take as input and what does it predict as output. The input data structure for this analysis is essay one response embedding, essay two response embedding, and the variable for prediction, here essay score 1 - essay score 2 score. Next, we create the essay pairs and the difference scores.
Essay pairing
The essay pairing is not entirely clear from the paper, but it appears to be created in the following way going by the text. Essays are processed in order and pairs are constructed between each essay and the next essay. Additionally, if pair A and B and pair B and C are included, pair A and C must be included too.
Consider the first 3 essays: essay 1 is paired with 2, essay 2 is paired with 3, so essay 1 is paired with 3. This helps the model learn transitive relations i.e., if essay 1 is better than 2 and 2 is better than 3 then the model needs to learn that 1 must be better than 3. Some pairings are omitted. Under this scheme, as we implement it, 1 is never paired with 4 for example.
Target and labels
The outcome that is predicted in this use case is the difference between the human scores for the paired essays (e.g., essay 1 score - essay 2 score). Although our reconstruction uses only one prompt, we followed the paper and normalised the relative score difference to allow the architecture to be used across prompts with different scoring ranges by dividing by the maximum possible score difference for the prompt. For inference the predicted differences were transformed back to the original scale before being added to the reference essay score to give the final prediction.
Model training
Essay pairs are fed through the encoder and multilayer perceptron model to predict the difference score in a forward pass. Next, the loss for this pair is calculated using Mean Squared Error (MSE). This loss is back propagated through the architecture, calculating gradients with respect to the loss for all model parameters.
We trained the psychometrics.ai implementation of the model as Xie et al (2022) did over five folds (train 60, test 20, validation 20). Within each fold, we ran 20 epochs as opposed to Xie et al.'s 80 epochs. Loss declined steadily to a near-floor value well before this stage in each fold. Validation set accuracy showed a mixed pattern across folds. Two of the five folds reached their best-validation checkpoint as early as epoch 2 and 7, but two others didn't reach theirs until epoch 17 and 18.
During optimisation weights are adjusted using gradient descent with AdamW. The adjustment size is determined by the learning rate. The paper states a single learning rate (1e-5) applied across the entire model, consistent with the authors' description and what we did too. We trained with repeated sequences of forward passes, back propagation, and optimization until convergence.
All models were run on Google Colab. The NPCR data and model doesn't need a large GPU, since BERT-base is small and batch size was only 5. For session stability, we found uploading a copy of the notebook rather than opening it from within Drive worked better and prevented sync conflicts which can cause session disconnects.
Model inference
Once the model was trained we moved on to predicting essay scores that have not been human scored (or more precisely here, for essays we held back for validation). To do so, we created a new data set of essays to feed through the trained model in a single forward pass. In actual scoring of new essays this step is called inference rather than training.
The key question here is what essay the new essay being scored should be paired with. The answer is that we can pair the essay to be scored with a randomly selected essay from the training data; although the paired essay should really be in response to the same prompt as the scored essay.
The authors report doing this multiple times selecting K random essays and using vote averaging to determine the predicted scores. We also used K=40 reference essays per prediction, matching the paper's recommendation.
Results and discussion
Training
Training loss dropped fast in the first two epochs and the mean across folds declined steadily to 0.006 by epoch 15 and 0.0055 by epoch 20, rather than the loss going flat. Continued improvements in late loss late suggests further training might close the gap with the authors reported QWK of .85 on this prompt.
Validation
Validation QWK climbed overall from a mean of 0.784 at epoch 1 to a peak of 0.806 at epoch 11, reaching 0.805 by epoch 20. Individual folds reached their best checkpoint at different points: two early (epoch 2 and epoch 7), one around epoch 12, and two late ts epoch 17-18. The late-training folds pull the mean up toward the end.
Test
The psychometrics.ai implementation achieved a mean QWK of .819 across the five folds. The gap to the paper reported result for this prompt of .85 is small and could close with small configuration changes or longer training. The best validation checkpoint occurred at different stages across folds, so there was no single stopping epoch that was optimal across folds.
While our implementation was developed from the paper rather than the authors’ released code (which differs from the published description in several implementation details) it achieved comparable accuracy. This suggests that the NPCR method can be independently implemented from the methodological description in Xie et al (2022).
Note on model deployment
One limitation of this method is that it does not produce a single deployment-ready model. It trains five separate models and reports their average performance. If the goal is deployment, rather than evaluation, a final model can be retrained on the full available training data using the hyper-parameters this initial work identified. Alternatively, an ensemble of the five models could be deployed, although this involves higher computational cost per scored essay.
References
Xie, J., Cai, K., Kong, L., Zhou, J., & Qu, W. (2022). Automated essay scoring via pairwise contrastive regression. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 2724-2733).
Hamner, B., Morgan, J., lynnvandev, Shermis, M., & Vander Ark, T. (2012). The Hewlett Foundation: Automated Essay Scoring. Kaggle. https://kaggle.com/competitions/asap-aes
Shermis, M. D. (2014). State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration. Assessing Writing, 20, 53-76.
Next page
Scoring: Autoregressive multi-trait scoring (ArTS)
Previous page
Neural end-to-end architectures
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
