How we test

What we do

We pick a plain question and ask many AI models the same thing, again and again.

For most tests we write the plan down before the first ask. The plan is logged, not locked, so if something changes later we say so on the test page. Each test page says whether a written plan is published.

Every ask is a new chat

Each ask starts from nothing. The model does not see its earlier answers or anyone else's. That is how we can ask the same question many times and see how much the answer varies.

What the numbers mean

A result is a count of answers. We count how often each answer came up, out of all the answers in the main run.

A script, not a person, decides how each answer is counted. Percentages are always shown next to the counts they come from.

The charts, the tables and the downloads use the same numbers.

Grey zone

A grey zone answer is a close call: the model was near going the other way and did not. It ended with a clear answer, but a second reader saw doubt in it, or the reply itself says the choice is hard or that the other choice would be fine.

Marking is separate from scoring. Marks never change a count or a result. Each mark says why, and where a wording rule found it, it shows the sentence.

Answers that did change, for example when the options were listed in the other order, are marked on their own.

Turning points and yellow lines

In the dilemma test, a model's answer turns from NO to YES as the group gets bigger. For each model, person, method and option order, the answer just before the turn and the answer just after it are marked as a turning point.

In each of those answers one line is marked in yellow. A person read the reply and picked the line that says the most: a principle, a burden the model says it carries, or an honest admission. That is a judgement, not a script, so it is shown as such and it changes no count.

We keep every answer

Nothing is deleted or edited. Failed asks and odd answers stay in the data. If a run is dropped, it is listed on the test page with the reason, and none of its answers are counted.

The raw files are the files our tools wrote when the test ran. The numbers on this site are recomputed from them every time the site is built, and the build stops if a total does not match.

One change we make to the raw files

Before publishing we take out details that are not about the answers: details about the computer that ran them, such as folder names and account limits, and billing details. Everything else is unchanged, and a fingerprint (SHA-256) of the untouched original is published next to each copy, so it can be checked.

Two ways of asking

OpenRouter sends one request per ask. Claude Code runs the model inside a command-line tool, which adds its own instructions around ours. The same model can behave differently on the two routes, so we never mix numbers from different routes in one ranking.

Limits

All tests so far were run over a few days, with a small number of tries per model. A small gap between two models may be luck. We do not claim a ranking.

Models change. A result describes the model as it answered on the day we asked.

We set very few options. Unless a test page says otherwise, settings such as temperature were left at each provider's default.

Runs we did not use

Some runs were stopped or replaced. They are kept and not counted.

• What is the greatest movie of all time?: A setup error sent the wrong question at the start of the first launch. It was stopped within seconds and none of its answers are used in any result.
• What is the greatest movie of all time?: A first run of the Claude models through Claude Code used an outdated version of the tool, so one model ran as an older version and another was unavailable. It was replaced by a run through OpenRouter and none of its answers are used.

Check our work

Every test page links to the JSON and CSV files and to the original raw files. The data page explains each field.