Scoring (1 of 3): Zero and Few-shot LLMs

In this first approach to transformer scoring of constructed responses we carry out zero shot and few shot scoring and compare it to human scores of the same constructed responses. I will create and share a scoring implementation for zero shot and few shot LLM scoring that replicates a section of a recent paper Jiang & Bosch (2024).

In particular, we will use OpenAI to score the free text responses in a classic education data set, Automated Student Assessment Prize (ASAP) -Automated Essay Scoring (AES) competition. This data set was released by the Hewlett foundation on Kaggle in 2012 and has become a benchmark data set in the field.

While the study reported by Jiang & Bosch did not achieve state of the art (SOTA) performance in score prediction on this data set it the study is nonetheless important for several reasons. The first reason it is important while not achieving SOTA is that the performance of these models is dependent on the quality of the rubrics and the rubrics with this data set are variable.

Given the moderate quality of rubrics, it is very possible that others are attaining higher performance in automated scoring using better developed rubrics. In IO psychology assessment, for example, the quality of the indicators in well designed assessment processes is usually higher. This method also does not need any training data unlike methods in other constructed response scoring categories we will explore.

Accessing the Kaggle data and our code

If you want to follow you'll need to visit Kaggle to get the data, as the rules are for the Kaggle competition that the data can’t be shared elsewhere. To get a copy of the data for yourself, visit the now closed competition, accept the competition rules and request a Kaggle token. You will need to use the token with the download syntax provided at the Kaggle size to download the zip from the command line on your local machine and then you can open the zip for the data files.

I chose to try to replicate results for essay 10, which is a grade level 8 science. This had 1640 scored responses for training and a hold out sample of 548 responses which were originally unscored as contestants needed to submit their predicted scores. As with LLM one shot and few shot there is not training, we focus here on simply LLM scoring a sample of the responses in the original file. We choose 100 examples here as the original study did. Experimentation costs quickly add up because the rubric and scored examples with each API call. To get the full experiment completed including trial and error was about $US40.

My code for the replication and the original code of Jiang and Bosch (2024) are both available on GitHub. The strategy I followed to code my version was to read the paper once to get to about reasonable level of understanding, look at the data file structures from Kaggle to understand variable names and file structures, and then use AI to assist with coding given what we are trying to do is relatively straight-forward.

This process turned out to be effective for achieving the baseline performance without scored examples but the approach was more temperamental at recovering the accuracy boost from including scored examples. Upon switching out my scored examples and replacing with the exact scored examples Jiang & Bosch provided but using my code and trying a few random seeds for the sampling I observed the boost they described (albeit not as strong). The authors themselves are cautious about their claims around the increased accuracy of examples.

Overall, I conclude that the effects they reported are reliable for this question as they are broadly replicable using different code on a different subsample of the data.

Process and code

Here the code used for the replication is presented in the core stages. This first code block imports required libraries, initializes the OpenAI client using an API key from the environment, verifies that the key is set, and confirms successful setup.

from openai import OpenAI
import pandas as pd
from sklearn.metrics import cohen_kappa_score
import numpy as np
import os

# Initialize client with new syntax
client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

# Check it's set
if client.api_key is None:
    raise ValueError("OPENAI_API_KEY environment variable not set!")
else:
    print("✓ API key loaded from environment")
    print(f"✓ Key starts with: {client.api_key[:8]}...")
print("Setup complete!")

This next code loads the TSV dataset into a pandas DataFrame and prints its size, column names, and the first row to inspect the data structure.

# Load the ASAP short answer dataset (TSV format)
df = pd.read_csv('train_rel_2.tsv', sep='\t')

# Basic info
print(f"Total responses: {len(df)}")
print(f"\nColumns: {df.columns.tolist()}")
print(f"\nFirst few column names:")
for col in df.columns[:10]:
    print(f"  - {col}")

# See the first row to understand structure
print(f"\n{'='*50}")
print(f"FIRST ROW (to understand data structure):")
print(f"{'='*50}")
print(df.iloc[0])

This code block filters the dataset to only Question 10, summarizes its score distribution, and prints one sample response for each score level. Note that Jiang and Bosch code ran across all essays so their code had sub-folder structuring to call data sets. Here this is not needed because we only study essay 10.

# Focus on Q10
q10 = df[df['EssaySet'] == 10]

print(f"{'='*50}")
print(f"QUESTION 10 DATA")
print(f"{'='*50}")
print(f"Total Q10 responses: {len(q10)}")
print(f"Score range: {q10['Score1'].min()} to {q10['Score1'].max()}")

# Score distribution
print(f"\nScore distribution:")
print(q10['Score1'].value_counts().sort_index())

# Show 3 sample responses (one for each score)
print(f"\n{'='*50}")
print(f"SAMPLE RESPONSES (one per score level):")
print(f"{'='*50}")

for score in sorted(q10['Score1'].unique()):
    sample = q10[q10['Score1'] == score].iloc[0]
    print(f"\n--- Score {score} Example ---")
    print(f"Response: {sample['EssayText'][:200]}...")  # First 200 chars
    print()

This code block stores the Question 10 prompt, rubric, and metadata as variables and prints a confirmation that the information was loaded. We use placeholders for the question which is available in the Kaggle data set.

# Official Question 10 information (PLACEHOLDERS USED, GET OFFICIAL QUESTION FROM KAGGLE)

Q10_GRADE = "<GRADE_LEVEL>"
Q10_SUBJECT = "<SUBJECT>"

Q10_QUESTION = """<INSERT OFFICIAL QUESTION PROMPT HERE>"""

Q10_RUBRIC = """<INSERT OFFICIAL SCORING RUBRIC HERE>"""

Q10_SCORE_RANGE = "<MIN_SCORE>-<MAX_SCORE>"

print("✓ Official Q10 information loaded")
print(f"✓ Grade: {Q10_GRADE}")
print(f"✓ Subject: {Q10_SUBJECT}")
print(f"✓ Score range: {Q10_SCORE_RANGE}")

This code defines (but does not yet run) a function that will later be called to send a prompt to GPT-4 and return the model’s predicted score.

def score_with_gpt4(prompt):
    """Call GPT-4 to score a response"""
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        max_tokens=10
    )
    return response.choices[0].message.content

print("✓ GPT-4 scoring function created!")

This code randomly stratifies 100 Q10 responses by score distribution, scores them with GPT-4 by calling the earlier function, computes accuracy and quadratic weighted kappa, and saves the results for later analysis. This gets us to the results for part one of the approach without scoring examples.

mport numpy as np
import pandas as pd
import random
import time

# Generate a fresh, truly random seed and record it
SEED = random.randint(1, 999999)
print(f"Testing 100 Q10 responses – EXACT 19-47-34 ratio (Jiang & Bosch)")
print(f"RANDOM DRAW – seed for this run: {SEED}  ← save this to reproduce exactly")

# Stratified sampling with the recorded seed
zeros = q10[q10['Score1'] == 0].sample(n=19, random_state=SEED)
ones  = q10[q10['Score1'] == 1].sample(n=47, random_state=SEED)
twos  = q10[q10['Score1'] == 2].sample(n=34, random_state=SEED)
test_sample = pd.concat([zeros, ones, twos]).sample(frac=1, random_state=SEED)

predictions_official = []
actuals_official = []

for idx, (_, row) in enumerate(test_sample.iterrows(), 1):
    prompt = create_prompt(row['EssayText'])  # NO examples
    pred = score_with_gpt4(prompt) 
    
    try:
        pred_int = int(pred.strip())
    except:
        print(f"Warning: Non-integer '{pred}' → defaulting to 0")
        pred_int = 0
    
    predictions_official.append(pred_int)
    actuals_official.append(row['Score1'])
    
    if idx % 10 == 0:
        print(f"Progress: {idx}/100 scored...")
    
    time.sleep(0.5)

# Metrics
from sklearn.metrics import accuracy_score, cohen_kappa_score
accuracy_official = accuracy_score(actuals_official, predictions_official)
qwk_official = cohen_kappa_score(actuals_official, predictions_official, weights='quadratic')

# Save results for plotting later
results_official_df = test_sample.copy()
results_official_df['gpt4_score'] = predictions_official
results_official_df['human_score'] = actuals_official
results_official_df['correct'] = results_official_df['gpt4_score'] == results_official_df['human_score']
results_official_df['error'] = results_official_df['gpt4_score'] - results_official_df['human_score']

# >>> MINIMAL ADDITIONS FOR PLOTTING <<<
results_official_df['condition'] = 'no_examples'
results_official_df['seed'] = SEED
results_official_df.to_csv(f"no_examples_seed_{SEED}.csv", index=False)
# >>> END MINIMAL ADDITIONS <<<

print(f"\n{'='*70}")
print(f"NO EXAMPLES – random seed {SEED}")
print(f"{'='*70}")
print(f"Accuracy: {accuracy_official:.3f} ({accuracy_official:.1%})")
print(f"QWK:      {qwk_official:.3f}")
print(f"{'='*70}")

Now we need to recreate the prompt but give it scoring examples for the comparison. Here I initially did not have success at getting the precision increase until I tried two things in combination (so it is not clear which drove the increase), added the precise scored examples used by the authors and tried a different random seed.

def create_prompt_with_official_examples(student_response):
    """Create prompt with official scoring examples from rubric documentation (paper-faithful)"""
    
    prompt = f"""You are a grader for a {Q10_GRADE}-grade {Q10_SUBJECT} exam in a high school. You will be provided with guidelines, scoring rubric, scoring examples, a question, and a student's response. Tables, if there are any, will be in CSV format. Rate the response according to the scoring rubric and scoring examples. You should reply to the response with rating followed by a paragraph of rationale. For the rating, just report a score only.

**Question:**
{Q10_QUESTION}

**Rubric:**
{Q10_RUBRIC}

**Rubric Range:** {Q10_SCORE_RANGE}

**Scoring Examples:**

0 points: "At nite when the dog is trying to sleep, having the paint color as black on the walls to make it darker on the inside of the house, when the dog is trying to sleep."
Notes: Color: Black. Effect on doghouse: At nite when the dog is trying to sleep, having the paint color as black on the walls to make it darker on the inside of the house when the dog is trying to sleep Not connected to the experiment. Results from experiment: None

1 point: "The white color i think will keep the doghouse cooler inside because if it was a dark color then it would conduct the heat and make it warm inside, but since it's white then it will reflect the heat."
Notes: Color: White. Effect on doghouse: ...will keep the doghouse cooler inside... OR ...because if it (the doghouse) was a dark color then it (the doghouse) would conduct the heat and make it (the doghouse) warm inside... Results from experiment: None

2 points: "The black paint will keep the inside of the dog house warm because a dark color will absorb more energy causing the temperature to increase. According to the data from Lid color vs. Air Temperature. Inside Glass jar, the darker the color, the greater the temperature would be inside the dog house."
Note: Color: Black. Effect on doghouse: …will keep the inside of the dog house warm because a dark color will absorb more energy causing the temperature to increase. Results from experiment: …the darker the color, the greater the temperature would be inside the dog house.

**Student Response:**
{student_response}

Score:"""
    
    return prompt

print("✓ Official examples prompt created (paper-faithful)")

Finally, we run the experiment again, using the exact same sample as we used in part 1 and using the scoring examples.

import time
import re
from sklearn.metrics import accuracy_score, cohen_kappa_score

# CRITICAL: Use the SAME seed as first run
print(f"Testing 100 Q10 responses WITH examples")
print(f"Using SAME seed as first run: {SEED}")
print(f"This ensures fair comparison\n")

# Recreate the SAME test_sample using stratified sampling
zeros = q10[q10['Score1'] == 0].sample(n=19, random_state=SEED)
ones  = q10[q10['Score1'] == 1].sample(n=47, random_state=SEED)
twos  = q10[q10['Score1'] == 2].sample(n=34, random_state=SEED)
test_sample = pd.concat([zeros, ones, twos]).sample(frac=1, random_state=SEED)

predictions_base = []
actuals_base = []

for idx, (_, row) in enumerate(test_sample.iterrows(), 1):
    prompt = create_prompt_with_official_examples(row['EssayText'])  # WITH examples
    
    max_retries = 3
    for attempt in range(max_retries):
        try:
            pred = score_with_gpt4(prompt)
            break
        except Exception as e:
            if "rate_limit" in str(e).lower() and attempt < max_retries - 1:
                wait_time = (attempt + 1) * 10
                print(f"Rate limit hit. Waiting {wait_time}s...")
                time.sleep(wait_time)
            else:
                print(f"Error on response {idx}: {e}")
                pred = "0"
                break
    
    try:
        match = re.search(r"Score:\s*([0-2])", str(pred))
        if match:
            pred_int = int(match.group(1))
        else:
            match = re.search(r"\b([0-2])\b", str(pred))
            pred_int = int(match.group(1)) if match else 0
        predictions_base.append(pred_int)
        actuals_base.append(row['Score1'])
        if idx % 10 == 0:
            print(f"Progress: {idx}/100 scored...")
    except Exception as e:
        print(f"Warning: Parsing failed for response {idx}: '{pred}' | {e}")
        predictions_base.append(0)
        actuals_base.append(row['Score1'])
    
    time.sleep(5)

accuracy_base = accuracy_score(actuals_base, predictions_base)
qwk_base = cohen_kappa_score(actuals_base, predictions_base, weights='quadratic')

# Save results for plotting
results_base_df = test_sample.copy()
results_base_df['gpt4_score'] = predictions_base
results_base_df['human_score'] = actuals_base
results_base_df['correct'] = results_base_df['gpt4_score'] == results_base_df['human_score']
results_base_df['error'] = results_base_df['gpt4_score'] - results_base_df['human_score']

# >>> MINIMAL ADDITIONS FOR PLOTTING <<<
results_base_df['condition'] = 'with_examples'
results_base_df['seed'] = SEED
results_base_df.to_csv(f"with_examples_seed_{SEED}.csv", index=False)
# >>> END MINIMAL ADDITIONS <<<

# ===== COMPARISON =====
print(f"\n{'='*70}")
print(f"RESULTS COMPARISON (seed {SEED}):")
print(f"{'='*70}")
print(f"WITH examples (run 2):")
print(f"  Accuracy: {accuracy_base:.3f}")
print(f"  QWK: {qwk_base:.3f}")
print(f"\nNO examples (run 1):")
print(f"  Accuracy: {accuracy_official:.3f}")
print(f"  QWK: {qwk_official:.3f}")
print(f"\nDifference between runs:")
print(f"  Accuracy: {accuracy_base - accuracy_official:+.3f}")
print(f"  QWK: {qwk_base - qwk_official:+.3f}")
print(f"\n{'='*70}")

Results

Panel A shows model accuracy improves from 0.710 (no examples) to 0.730 (with examples), showing a small performance gain from adding examples.

Panel B shows Quadratic Weighted Kappa increases from 0.733 to 0.750 with examples, indicating better agreement with human graders.

Panel C shows when examples are included in the prompt the model makes fewer big scoring errors (e.g., 0 when the true score is 2) even though small errors still occur at similar rates.

Panel D shows correct predictions dominate the diagonal in the confusion matrix without examples, but the model frequently confuses adjacent scores.

Panel E shows that with examples, the fiagonal accuracy increases slightly, especially for score 1 and score 2, indicating improved classification with examples.

Panel F is a score-shift plot that shows most responses retain the same score across conditions (largest bubbles on the diagonal), with some shifts when examples are added.

References

Jiang, L., & Bosch, N. (2024, July). Short answer scoring with GPT-4. In Proceedings of the Eleventh ACM Conference on Learning@ Scale (pp. 438-442).

Shermis, M. D., & Hamner, B. (2012). Contrasting state-of-the-art automated scoring of essays: Analysis. In: Paper presented at the National Council of Measurement in Education.

Shermis, M. D., & Hamner, B. (2013). Contrasting state-of-the-art automated scoring of essays. In: M. D. Shermis & J. Burstein (Eds.).

The Learning Agency Lab. (n.d.). Automated Student Assessment Prize (ASAP) dataset. Kaggle.

Next page

Ground truth in AI psychometrics

Previous page

Crash course in transformer-era automated scoring

Return home

Psychometrics.ai

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).