Deep20Bench 1.1 · Current edition
Deep20Bench is a public AI benchmark for large language models (LLMs). It measures how many questions a model needs to identify a hidden subject: a person, character, or thing. Most questions get a clear Yes or No. When the facts are less clear, the answer can be Rather yes, Rather no, or Unknown.
How scoring works
The model only knows a broad category, such as “video game character”. It gets up to 40 questions, more than the traditional twenty, so more rounds can finish. Each question, wrong guess, or reply in the wrong format adds one to the score. If the model never finds the subject, the round scores 41. The final score is the average over all rounds; lower is better.
This edition
This edition compares 21 models on 10 subjects, with 3 rounds per subject. All results and transcripts are public. Each edition has its own comparison settings and results.
Suggest what we should test next on GitHub Discussions.
What the game tests
Each answer should improve the model’s next question. Why this game works →
Know which facts help narrow down the subject.
Choose questions that remove many possibilities.
Use all earlier questions and answers to plan the next question.
Guess at the right moment, before the limit.
How the game is played
The Guesser is the model being tested: it asks questions and makes the final guess. The Oracle answers using web research and, if the run's rules allow, its own knowledge. A Reviewer checks every answer except Unknown without seeing the Oracle's answer. If they disagree, a Judge decides without seeing either answer. The Guesser receives only the final answer.
How the game and answer checks work →Edition 1.1 results
Lower is better. A failed round scores 41.
First place
This model used 410 questions over 30 rounds (10 subjects × 3) and solved every round.
Same average score as Gemini 4 Argon (high)Rank 2; ranked higher by narrower 95% CI (2.36 vs 2.56).
Round scores are averaged for each subject, then across subjects. A failed round gets the fixed failure score.
The 95% confidence interval shows how certain the average is, based on repeated rounds on these subjects. A wider interval means less consistent results. It does not predict scores on other subjects.
Models rank by exact average score, then by narrower 95% confidence interval (CI). Equal scores and CI widths share a rank.
Lower is better. The blue marker is the average question score. The line is its 95% confidence interval (CI); darker lines are wider. The side plot shows each CI width in three equal bands.
| Rank | ||||||
|---|---|---|---|---|---|---|
| Rank 1 | GPT-6 Astra (high) | Question score13.67questions · lower is better12.49–14.85 | High | 100% | 100% | $6.68 |
| Rank 2 | Gemini 4 Argon (high) | Question score13.67questions · lower is better12.39–14.95 | High | 100% | 99.8% 1 violation | $5.34 |
| Rank 3 | Gemini 3.8 Flash (high) | Question score14.87questions · lower is better13.19–16.54 | High | 100% | 100% | $4.52 |
| Rank 4 | Claude Opus 5.5 (high) | Question score15.00questions · lower is better13.24–16.76 | High | 97% | 100% | $3.84 |
| Rank 5 | Claude Fable 5.1 (high) | Question score15.10questions · lower is better13.76–16.44 | High | 93% | 99.4% 3 violations | $10.22 |
| Rank 6 | GPT-6.1 Sol (high) | Question score15.23questions · lower is better12.84–17.62 | High | 97% | 100% | $2.87 |
| Rank 7 | Gemini 3.7 Flash (high) | Question score16.57questions · lower is better14.35–18.79 | High | 93% | 100% | $3.80 |
| Rank 8 | Grok 4.6 (high) | Question score17.13questions · lower is better15.54–18.73 | High | 100% | 100% | $6.56 |
| Rank 9 | Claude Sonnet 5.5 (high) | Question score17.83questions · lower is better15.49–20.18 | High | 90% | 100% | $3.48 |
| Rank 10 | GPT-6 Sol (high) | Question score18.03questions · lower is better16.29–19.77 | High | 100% | 100% | $3.35 |
| Rank 11 | Claude Opus 5 (high) | Question score18.07questions · lower is better16.50–19.63 | High | 93% | 99.6% 2 violations | $5.55 |
| Rank 12 | GPT-5.6 Sol (high) | Question score18.07questions · lower is better14.38–21.75 | High | 97% | 100% | $4.27 |
| Rank 13 | Muse Spark 1.3 Contributor (high) | Question score18.37questions · lower is better16.61–20.13 | High | 97% | 100% | $2.60 |
| Rank 14 | Muse Wonder (high) | Question score19.63questions · lower is better17.58–21.68 | High | 97% | 100% | $7.18 |
| Rank 15 | Claude Haiku 5.5 (high) | Question score20.00questions · lower is better17.10–22.90 | High | 83% | 100% | $2.77 |
| Rank 16 | GPT-5.6 Luna (high) | Question score21.03questions · lower is better18.51–23.56 | High | 90% | 100% | $3.38 |
| Rank 17 | GPT-6 Luna (high) | Question score21.17questions · lower is better18.18–24.15 | High | 80% | 100% | $2.63 |
| Rank 18 | Grok 4.7 (high) | Question score22.23questions · lower is better20.80–23.67 | High | 73% | 100% | $4.09 |
| Rank 19 | gpt-oss-120B (high) | Question score22.23questions · lower is better19.58–24.89 | High | 67% | 89.3% 71 violations | $4.57 |
| Rank 20 | GLM-5.3-Flash (high) | Question score23.27questions · lower is better19.58–26.96 | High | 73% | 93.2% 49 violations | $2.88 |
| Rank 21 | MiniMax M3 (high) | Question score29.17questions · lower is better24.71–33.62 | High | 50% | 99.1% 8 violations | $4.35 |
How to read the results
We test a small, fixed set of subjects. Every game record is public, so you can check the results. What the results can show →
Models in each edition use the same subjects, number of rounds, question limit, and scoring rules.
A failed round gets a fixed score. Before the question limit, a reply in the wrong format uses up a turn.
Each model's results link to every subject, round, question, answer, source, cost, and time.
Origin
Patrick Heusser and Markus Tuor came up with the idea while playing Twenty Questions with the kids. Patrick then designed and built the project.