Method · Edition 1.1
How Deep20Bench works
Deep20Bench measures how many questions an AI model needs to find a hidden subject. Each model plays the same subjects several times. The fewer questions it needs, the better its score. A round where it never finds the subject gets a fixed penalty.
01 · The game
Find a hidden subject.
The model under test, called the Guesser, must identify a hidden subject: a person, character, or thing. It starts with only a broad category, asks one question at a time, and guesses when it is ready.
The Guesser has up to 40 counted questions, then one final guess. See what counts toward the score.
What the answers mean
Most questions get a clear Yes or No. The other answers cover questions the facts do not settle clearly.
- Yes
- The information supports a clear Yes.
- Rather yes
- It points toward Yes, but an important gap remains.
- Rather no
- It points toward No, but an important gap remains.
- No
- The information supports a clear No.
- Unknown
- There is no clear direction, or unclear wording or conflicting evidence could change the answer.
Not finding something in a search is not enough for No or Rather no. Rather yes and Rather no are clues, not probabilities, and they do not say how often something is true. The Guesser should keep other possibilities open and check important assumptions with another question.
Guess rules and the question limit
Guesses get only Yes, No, or Unknown. Only Yes ends a round successfully.
A guess must name the subject exactly. If the subject is a general kind of thing, a recognized subtype also counts if it keeps the defining features and meets the subject’s stated limits. Extra detail does not make a correct guess wrong.
The limit is higher than twenty so that a model that needs, say, 25 questions scores better than one that never finds the subject. Every extra question still raises the score. At the limit, the model can only guess. If that guess is wrong, the round scores 41, one more than any successful round.
02 · Answer checks
How answers are checked.
Other models know the hidden subject and answer the Guesser’s questions. Up to three of them handle each question, each with a separate job:
Oracle: find an answer.
Searches for evidence and proposes an answer. Unknown is final; every other answer needs a second check.
Reviewer: check independently.
Checks the question without seeing the Oracle’s answer. If they agree, that answer is final.
Judge: resolve disagreement.
Decides whenever their answers differ, including Reviewer Unknown. It sees neither earlier answer.
The second check exists because a single answering model can make basic mistakes. In early runs, one Oracle said Yes to “born before 1800?” while citing 1875. The Reviewer and Judge use different model families and providers to reduce shared mistakes.
Guesses go to a separate Guess Validator, which checks whether the guess names the subject. The Guesser receives only the final answer, with no explanation.
A repeated question can reuse an earlier, fully checked answer. The transcript marks every reused answer.
What the checking models can see
Oracle: the subject and the current question, but no earlier turns. It searches the web and keeps the source text. Under current rules, it may also use its own knowledge, with a short private justification.
Reviewer and Judge: the subject, the question, and the Oracle’s numbered evidence. They cannot search the web or see earlier answers or the game history. Any difference, even Yes versus Rather yes, goes to the Judge.
In current runs, the Reviewer and Judge may also use their own knowledge, with a short private justification. An earlier accepted version limited the Reviewer to the supplied evidence and let the Judge use its own knowledge only in limited cases. Each run records its rules and prompt versions.
Guess Validator: only the subject and the guess, with no web access.
If a required check fails, the round counts as a test system failure; the Oracle’s answer is never used on its own.
The Guesser never sees sources, other models’ instructions, checking decisions, logs, or private files. Its own instructions and format reminder contain nothing about the subject.
When an earlier answer is reused
Benchmark runs reuse answers by default. A question that matches an earlier one can reuse its fully checked answer, from an earlier compatible round or from the same game. The run’s recorded rules decide which answers qualify.
A reused answer keeps its original evidence, time, and source label. No new search or check runs. The question still counts, but adds no answer-checking cost. The Guesser sees only the final answer. Compare runs only if they use the same reuse rules and answer history.
03 · The score
Fewer questions means a better score.
Each round gets a score. Average the rounds for each subject, then average the subjects. Every subject has equal weight. The result is the model’s question score.
- Each question
- +1
- Each wrong guess
- +1, play continues
- Each reply in the wrong format
- +1, play continues; an invalid final guess ends the round instead
- Correct guess
- Not counted; the round ends
- Subject not found
- The round scores 41
- Test system failure
- Not scored; the run stays incomplete until the round is rerun
For example, 12 questions, one wrong guess, and one reply in the wrong format before the correct guess score 14.
Each score has a 95% confidence interval (CI). It estimates how much the average could change if the same subjects were played again. A narrower interval means more consistent results, even if the score itself is poor. It does not describe performance on new subjects.
The Stability view ranks models by this interval’s width. Cost and time are reported separately and do not change the question score.
Models rank by exact average score, then by narrower 95% confidence interval (CI). Equal scores and CI widths share a rank. This tie-breaker is a ranking rule, not a test of whether one model is significantly better.
Format errors and failed rounds
Each Guesser reply must be one valid ASK or GUESS action. Before the question limit, an invalid reply uses one counted turn and gets a fixed FORMAT_ERROR reminder. The reminder says nothing about what went wrong, whether the model was right, or the subject. Too many invalid replies in a row end the round as a scored failure. An invalid final guess ends the round with no retry and no extra counted turn.
A later correct guess does not erase earlier format errors. Result pages show the share of valid replies, the number of errors, and the affected rounds. The used turn is already in the question score; there is no second penalty.
Scoring formulas and uncertainty
Score policy: average-then-average-v1.
How the interval is calculated
Estimate the variation between rounds separately for each subject, divide by that subject’s round count, then combine the estimates with equal subject weights. A Welch-Satterthwaite t interval allows different variation for each subject.
The interval describes uncertainty in the average, not the range of single rounds. It also does not cover changes to models, providers, or editions. Comparing two models’ intervals is not a significance test.
The interval assumes that rounds on the same subject are independent. Different seeds support this but do not prove it, and reused answers can link rounds. The interval does not adjust for this or estimate how new web research might change answers.
04 · Comparisons
Repeat the game on the same subjects.
In this edition, each model plays 10 subjects 3 times each: 30 rounds in total. Every round starts a fresh Guesser conversation. Repetition shows how much results vary.
Only complete runs with accepted settings enter the leaderboard. If several runs qualify for a model, the newest one counts, never the best one.
Edition 1.1 accepts a few reviewed versions of the answer-checking prompts and subject descriptions, so runs may differ in more than the Guesser. These differences can affect scores. Each run shows its settings.
Subjects, repeated rounds, and future editions
Each subject has a main name, accepted alternative names, a clear description, and a public reference. Every model in an edition plays the same list, shown with categories in each run. The selection is not random, balanced, or representative. Cost limits its size: each added subject needs repeated model turns and answer checks.
A round cannot see another round’s transcript, evidence, or private data. Each round starts with a code made from the base seed and the round number. It is the same for every model and subject and reveals nothing about the subject. Base seed: 0.
Future editions should include more subjects and kinds, such as places and objects. Subjects and selection rules will be fixed before testing. Any change to subjects or game rules starts a new edition with separate results.
Requirements for a published run
Edition 1.1 fixes the subjects, rounds per subject, question limit, game rules, and scoring rules. A run must also match one accepted release in full: prompt versions, subjects, seed, game rules, answer-checking model settings, and saved check records. Other versions and incomplete test runs do not qualify. Published runs keep their experiment label if they have one.
- Signed run files must pass integrity checks. Any invalid file stops the site build.
- The run must be finished, with every round of every subject complete.
- Rounds the model failed count as scored. A run with missing rounds or test system failures stays out until those rounds are rerun.
- For a rerun round, only the final attempt counts toward scores and cost comparisons. Earlier attempts stay in the records and the total spend.
05 · Explore results
Follow a score back to the game.
Open a model’s run to see its subjects and rounds. Each round shows the questions, answers, guesses, and supporting evidence, along with format errors, usage, cost, and timing.
Answers are checked by models, not independently verified. The transcripts let you check the evidence and spot mistakes.
For background on how it was built, read the story on Medium ↗.
What changed between editions?Compare editions 1.0 and 1.1
Benchmark editions
What changed in 1.1
In edition 1.0, questions get Yes, No, or Unknown. Edition 1.1 adds Rather yes and Rather no, so an answer can lean one way when the evidence is incomplete.
| Setting | Edition 1.1 | Edition 1.0 |
|---|---|---|
| Question answers | Yes, Rather yes, Rather no, No, Unknown | Yes, No, Unknown |
| Subjects | 10 | 7 |
| Rounds per subject | 3 | 5 |
| Question limit | 40 | 50 |
| Failure score | 41 | 51 |
| Review rules | Evidence or model knowledge, as each run records; any disagreement goes to the Judge | Evidence first; model knowledge only for settled facts |
| Guess validation | Yes, No, Unknown | Yes, No, Unknown |
Why add Rather yes and Rather no?
Web evidence often points one way without fully proving the claim. With only three answers, Unknown throws that hint away, while a firm Yes or No can claim more than the evidence shows. The two new answers keep the hint and mark it as uncertain. Whether this improves the game still needs testing.
Edition 1.1 also revises the Guesser's strategy instructions and adds a fixed guide to what each category means. Both editions use the same scoring formula, but results are calculated separately. Because the subjects, round counts, limits, and prompts differ, a score difference between editions does not show the effect of the new answers or that a model improved.