Origin · news · prior work
Patrick Heusser and Markus Tuor came up with Deep20Bench while playing Twenty Questions with the kids. Patrick then designed and built the benchmark.
Updates
New models, releases, and articles. A score is the average number of questions per round, with a fixed penalty for failed rounds. Lower is better.
It averaged 20.00 questions per round in edition 1.1, 15th of 21 models, with 5 failed rounds out of 30. Compared with GPT-6 Luna (high) at 21.17, it needed fewer questions on seven of the ten subjects, with clear leads on Garfield and Achilles. Spider web was its weak spot: 30, 38, and 21 questions, against Luna’s 16, 22, and 15.
It averaged 13.67 questions per round in edition 1.1, the same as GPT-6 Astra (high): both used 410 questions over 30 rounds and solved every round. Astra ranks first because its 95% confidence interval is narrower. Argon was the best of 20 models on Door handle and second on Albert Schweitzer and Bike pump, but 19th on Moon, which took it 11, 12, and 14 questions.
It averaged 19.63 questions per round in edition 1.1, 13th of 19 models at the time. Some subjects were much harder for it than others: it was 9th best on Albert Schweitzer but 18th on Garfield, which took it 10, 22, and 30 questions. Achilles took 8 questions every time.
It averaged 17.83 questions per round in edition 1.1, with 3 failed rounds out of 30.
It averaged 15.23 questions per round in edition 1.1, with 1 failed round out of 30.
It averaged 15.00 questions per round in edition 1.1, with 1 failed round out of 30.
It averaged 18.03 questions per round in edition 1.1, with no failed rounds.
It averaged 21.17 questions per round in edition 1.1, with 6 failed rounds out of 30.
It averaged 22.23 questions per round in edition 1.1, with 8 failed rounds out of 30.
It averaged 15.09 questions per round in edition 1.0, with 2 failed rounds out of 35.
It averaged 13.14 questions per round in edition 1.0, with no failed rounds.
It averaged 12.11 questions per round in edition 1.0, with 1 failed round out of 35.
Patrick writes about what it took to build Deep20Bench: wrong answers from the answering model, the blind second check, format errors, and the design choices along the way.
An anonymous stealth model on OpenRouter. It averaged 17.60 questions per round in edition 1.0, 12th of 15 models at the time.
It averaged 14.03 questions per round in edition 1.0, with 1 failed round out of 35.
It averaged 14.29 questions per round in edition 1.0, with no failed rounds.
It averaged 12.06 questions per round in edition 1.0, with no failed rounds.
Prior work
Deep20Bench was developed independently. These projects work on related problems.
| Year | Focus | Publication | Source |
|---|---|---|---|
| 2018 | Question strategy | Learning-to-Ask: Knowledge Acquisition via 20 QuestionsLearns which questions to ask to identify a subject and gather knowledge. | Microsoft Research ↗ |
| 2022 | World knowledge | 20Q: Overlap-Free World Knowledge Benchmark for Language ModelsUses Twenty Questions to test world knowledge with separate training and test subjects. | ACL Anthology ↗ |
| 2024 | Planning benchmark | The Entity-Deduction ArenaTests how language models plan questions and use earlier answers to find a hidden subject. | Apple Machine Learning Research ↗ |
| 2025 | Choosing useful questions | Adaptive Elicitation of Latent Information Using Natural LanguageChooses the questions that most reduce uncertainty about the answer. | OpenReview ↗ |
| 2025 | Information gain | BED-LLM: Intelligent Information Gathering with LLMsChooses questions by how much information each answer is expected to give. | Apple Machine Learning Research ↗ |
Next
Read the rules or inspect the current model runs.