Why Deep20Bench
Easy to play, hard to play well.
Anyone can play Twenty Questions. Playing it with few questions is hard. That makes it a simple, repeatable test of how well AI models use what they know.
01 · What it tests
Why the game works as a test.
The game is uneven.
The Oracle knows the subject and only checks one fact per question. The Guesser starts with almost nothing and must search a huge space. Checking one fact about a known subject is far easier than finding a good question without knowing it. Answer mistakes still happen, so answers get a second check.
Each question must build on the last.
The first questions are easy: real or fictional, alive or dead. After that, broad questions stop helping. The best next question uses every earlier answer and splits the remaining possibilities roughly in half. Twenty perfect splits can separate about a million possibilities (220).
Knowledge alone is not enough.
Recognizing Albert Einstein in a photo is easy. Finding him with only Yes and No answers is not. There is no universal list of questions: after a good opening, the right question depends on what the model knows about the candidates that are left.
Simple, repeatable, and extendable.
The result is one number, the questions needed, measured the same way for every model. Adding a subject to the catalog takes only a name, a description, and a public reference; testing it still takes repeated rounds and answer checks. New editions can use new subjects.
It also tests discipline.
Every reply must follow a fixed format for dozens of turns. Some models follow it for ten turns, break it once, and recover after one reminder. Others forget the rules in the middle of a game. Before the question limit, each broken reply costs a turn.
02 · Limits
What it does not show.
The game measures a narrow mix of skills: world knowledge, planning, using earlier answers, and following rules. It is not a measure of general intelligence. Each edition uses a small, fixed set of subjects, and other models check the answers, so mistakes are possible. The subjects and transcripts are public, so a later model may have seen them in training; an advantage from training data cannot be ruled out.
See how answers are checked and what a comparison can tell you.