Data
Every test comes as one JSON file and one CSV file, and the original raw files are here too. All of it is free to download.
| Test | Answers | JSON | CSV | Original files |
|---|---|---|---|---|
| What is the greatest movie of all time? | 437 | JSON 289 KB | CSV 144 KB | Fingerprints |
| Who is the smartest person who ever lived? | 494 | JSON 330 KB | CSV 167 KB | Fingerprints |
| Does telling an AI to think harder help it avoid the letter e? | 60 | JSON 69 KB | CSV 35 KB | Fingerprints |
| At what size of group does an AI sacrifice one person to save the group? | 512 | JSON 808 KB | CSV 800 KB | Fingerprints |
| Can an AI write three lines without the letter e? | 30 | JSON 35 KB | CSV 17 KB | Fingerprints |
What is in each file
Every answer is one record. These are its fields.
- id
- A stable name for the answer.
- model_name
- The full name of the model.
- model_id
- The exact model id that was sent.
- route
- How the model was called: OpenRouter or Claude Code.
- served_by
- Who actually served the answer, when it is known.
- phase
- The part of the test: trial run, main run, or asked with a reason.
- prompt_id
- Which prompt was asked.
- run_index
- The number of the ask for this model and prompt.
- condition
- What was varied in this test, for example the group size.
- ok and error
- Whether the ask worked, and why not if it failed.
- text
- The full answer, exactly as returned.
- verdict
- How our script read the answer: a label, a value and flags.
- ts_start and ts_end
- When the ask was sent and when it finished, when known.
- tokens
- Input, output and thinking tokens, when known.
- latency_s
- Seconds the model took.
- source
- The published raw file and the line in it this answer came from.
- grey
- Why the answer is in the grey zone (it hesitated but did not change), when it is: a list of reasons.
- readers
- What a second reader made of the reply, when that is known.
- changed
- Why the answer counts as a change of mind, when it does: a list of reasons.
- grey_notes
- The sentence from the reply that triggered a wording mark, when there is one.
- turning
- Marks an answer next to a turning point: before (the last NO) or after (the first YES).
- highlight
- The line picked as the most telling in a turning-point answer, by reading.
About the CSV files
A text cell that starts with =, +, - or @ gets one apostrophe in front, so spreadsheet programs do not run it as a formula. The JSON files are exact.
The fingerprint list next to each set of raw files gives the SHA-256 of the original file, so you can check a copy against ours.