[Python] OpenAI GPT-3 token logprobs and word-level surprisal: inconsistent values and missing...

I’m trying to compute word-level surprisal values for a set of sentence stimuli using OpenAI’s Completions API (legacy endpoint).

In information-theoretic terms, surprisal is the negative base-2 logarithm of a word’s conditional probability given its preceding context.

For example, in:

1. I take my coffee with cream and dog the word “dog” should yield a high surprisal (because unexpected/unpredictable), while in:
2. I take my coffee with cream and sugar “sugar” should yield a low surprisal because it’s a more predictable completion.

• I’m using the Completions API via client.completions.create() with echo=True and logprobs=1, so I can access token-level logprobs for the input.
• My stimuli are stored in a .txt file with the target word flanked by asterisks (e.g. make-up).
• I then: 1. Replace the asterisks with a rare marker (α) to locate the target span. 2. Tokenize the whole sentence using the GPT-2 tokenizer (to match OpenAI’s model). 3. Extract the logprobs for the tokens in the target span. 4. Sum them, then convert from nats to bits (-np.sum(logprobs) / np.log(2)).

This current script is adapted from a published version, which was optimised for use with single-token target words: https://osf.io/hqwer

I don't think my script handles multi-token target words well... (see Issues below) See below for the script, what currently seems to work and what doesn't:

import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"
import argparse
from transformers import AutoTokenizer
import numpy as np
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def parse_args():
parser = argparse.ArgumentParser(description='Calculates surprisal and other \
metrics (in development) of transformers language models')

parser.add_argument('--stimuli', '-i', type=str,
help='Stimuli to test.')
parser.add_argument('--stimuli_list', '-ii', type=str,
help='Path to file containing list of stimulus files to test.')
parser.add_argument('--output_directory','-o', type=str, required = True,
help='Output directory.')
parser.add_argument('--model','-m', type=str,
help='The name of the GPT-3 model to run.')
parser.add_argument('--model_list','-mm', type=str,
help='Path to file with a list of GPT-3 models to run.')

args = parser.parse_args()
return args

def process_args(args):

try:
output_directory = args.output_directory
except:
print("Error: Please specify a valid output directory.")

if not os.path.exists(output_directory):
try:
os.makedirs(output_directory)
except:
print("Error: Cannot create output directory (Note: output directory does not already exist).")

if args.model_list:
try:
assert os.path.exists(args.model_list)
with open(args.model_list, "r") as f:
model_list = f.read().splitlines()
except:
print("Error: 'model_list' argument does not have a valid path. Trying to use individual specified model.")
try:
assert args.model
model_list = [args.model]
except:
print("Error: No model specified")
else:
try:
assert args.model
model_list = [args.model]
except:
print("Error: No model specified")


if args.stimuli_list:
try:
assert os.path.exists(args.stimuli_list)
with open(args.stimuli_list, "r") as f:
stimulus_file_list = f.read().splitlines()
except:
print("Error: 'stimuli_list' argument does not have a valid path. Trying to use individual stimulus set.")
try:
assert args.stimuli
stimulus_file_list = [args.stimuli]
except:
print("Error: No stimuli specified")
else:
try:
assert args.stimuli
stimulus_file_list = [args.stimuli]
except:
print("Error: No stimuli specified")

return(output_directory,model_list,stimulus_file_list)

def run_models(output_directory, model_list, stimulus_file_list):
tokenizer = AutoTokenizer.from_pretrained("gpt2")

for j in range(len(model_list)):
model_name = model_list[j]
model_name_cleaned = "gpt3" + model_name.replace("-", "_")

for i in range(len(stimulus_file_list)):
stimuli_name = stimulus_file_list.split('/')[-1].split('.')[0]
filename = f"{output_directory}/{stimuli_name}.surprisal.{model_name_cleaned}.causal.output"

with open(filename, "w") as f:
f.write("FullSentence\tSentence\tTargetWords\tSurprisal\tNumTokens\n")

with open(stimulus_file_list, 'r') as f:
stimulus_list = f.read().splitlines()

for j, stimulus in enumerate(stimulus_list):
try:
stimulus_spaces = stimulus.replace("*", "α").replace(" α", "α ")
encoded_stimulus = tokenizer.encode(stimulus_spaces)

if len(tokenizer.tokenize("aα")) == 2:
dummy_var_idxs = np.where(
(np.array(encoded_stimulus) == tokenizer.encode("α")[-1]) |
(np.array(encoded_stimulus) == tokenizer.encode("aα")[-1])
)[0]
preceding_context = encoded_stimulus[:dummy_var_idxs[0]]
target_words = encoded_stimulus[dummy_var_idxs[0] + 1 : dummy_var_idxs[1]]
following_words = encoded_stimulus[dummy_var_idxs[1] + 1 :]

stimulus_cleaned = stimulus.replace("*", "")

output = client.completions.create(
model=model_name,
prompt=stimulus_cleaned,
max_tokens=0, # Ensure we don't generate extra text
temperature=0,
n=1,
stream=False,
logprobs=1,
echo=True,
stop="\n",
)

print(f" API Response for line {j+1}: {output}")

logprob_data = output.choices[0].logprobs

# token-level log probabilities
tokens = logprob_data.tokens
logprob_tokens = logprob_data.token_logprobs

if logprob_tokens is None or all(lp is None for lp in logprob_tokens): # Only flag if ALL logprobs are missing
print(f"❌ Missing logprobs for line {j+1}: {stimulus_list[j]}")
continue # Skip this stimulus if all logprobs are missing

# Extract only the target word's logprobs
surprisal_list = logprob_tokens[len(preceding_context) : len(preceding_context) + len(target_words)]
surprisal_list = np.nan_to_num(surprisal_list, nan=0.0) # Handle missing values

# Correct surprisal calculation
sentence = tokenizer.decode(preceding_context + target_words)
target_string = "".join(tokens[len(preceding_context) : len(preceding_context) + len(target_words)])
print(f"target words: {target_words}")
surprisal = -np.sum(surprisal_list) / np.log(2)
num_tokens = len(target_words)

with open(filename, "a") as f:
f.write(f"{stimulus.replace('*','')}\t{sentence}\t{target_string}\t{surprisal}\t{num_tokens}\n")

except Exception as e:
print(f"❌ Problem with stimulus on line {j+1}: {stimulus_list[j]}")
print(f" Error: {e}") # Print the exact error message from OpenAI
print(f" OpenAI Response: {output}") # Show what OpenAI actually returned

def main():
args = parse_args()


try:
output_directory, model_list, stimulus_file_list = process_args(args)
except Exception as e:
print(f"❌ Error: {e}")
return False

try:
run_models(output_directory,model_list,stimulus_file_list)
except:
print("Error: issue with stimuli, output directory, GPT-3 models chosen, or API key.")



if __name__ == "__main__":
main()


The script runs without any exceptions. It (seemingly) correctly extracts and processes many target words...

⸻

Issues:

1. Wrong token extracted - In a few cases, the wrong token is identified as the target. For example, in: '... the airport...' the output logs surprisal for "the" instead of "airport", despite the target being marked correctly in the input.

Could this point to an issue with how the script calculates token offsets after replacing the * markers with α ?

2. Missing values: - Some target words return a surprisal of -0.0, even when the tokens appear correctly identified.

3.. Counterintuitive values and multi-token handling: - In some multi-token cases, surprisal is higher in the plausible sentence than in the implausible one. For example:

1. Jane put some make-up on. # Higher surprisal (!?)
2. Phil put some make-up on. # Lower surprisal

This contradicts what we’d expect — that make-up should be more predictable in sentence 3.

• The original version of this script assumed single-token targets. Many of my targets are multi-token (eg "make-up", "beak", "chocolate") and may not be handled properly..

Any advice would be hugely appreciated — especially from anyone who's worked with OpenAI token logprobs or surprisal modelling using these APIs. Thank you!

Continue reading...