Deep20Bench
v1.1Current

Benchmark edition

Edition 1.1 settings

10 subjects3 rounds per subject40 question limit
Question answersYesRather yesRather noNoUnknown
What changed
Source
Menu
Run workspace

Official run

Gemini 4 ArgonHigh

In this Twenty Questions AI benchmark, Gemini 4 Argon (high) ranks 2 of 20 with a question score of 13.67: the average number of questions per round, where lower is better. It solved every round. The run has 30 scored rounds: 10 subjects, 3 rounds each. Choose a subject to see its rounds.

of 20google-vertexHigh

Question score13.67questions per roundLower is better · 95% CI 12.39–14.95
Solved
100%
Valid format
99.8%
Guesser cost
$3.94
Guesser time
1h 6m

Drill down

Subjects.

01Albert Einstein10.67 avg02Albert Schweitzer16.33 avg03Garfield12.00 avg04Achilles8.67 avg05Genghis Khan11.67 avg06Bike pump22.67 avg07Spider web13.33 avg08Eyebrow13.00 avg09Moon12.33 avg10Door handle16.00 avg

Complete benchmark

Run totals.

Includes the Guesser and all answer-checking models.

Total tokens
3,280,848
Total runtime
2h 18m
Guesser calls
440
Completed

Reply format

Format errors found.

1 invalid reply in 1 round used up 1 counted turn.

Run details

Execution
BX-20261014-B-0003-official-M0034-002
Model ID
M-0034
Benchmark
B-0003
Base seed
0
Git commit
237889d972fe

Recorded cost

By role.

Cost composition.Share of full-run cost
  1. Guesser73.7%$3.94
  2. Oracle13.6%$0.73
  3. Reviewer7.7%$0.41
  4. Judge4.8%$0.26
  5. Validator0.1%$0.01
Rerun cost (excluded)$0.202 earlier attempts in 2 rerun rounds. Not included above.

Roles in this cost:

Guesser
The tested model. Asks questions and makes guesses.
Oracle
Searches for evidence and proposes an answer.
Reviewer
Checks each Oracle answer except Unknown, without seeing it.
Judge
Decides when the Oracle and Reviewer disagree.
Validator
Checks whether a guess names the hidden subject.
How the roles and answer checks work →

Run configuration

Models.

Models, prompt versions, providers, calls, and cost for the whole run.

GuesserModel under test

google/gemini-4-argon

The tested model. Asks questions and makes guesses.

Calls
440
Cost
$3.9410
Reasoning
High
Routing
Fixed provider · google-vertex
Resolved model
google/gemini-4-argon
Resolved provider
Google
Prompt version
stateful-category-guesser-v16-five-answer-category-guide
Configuration
M-0034
Provider details
Google440 calls$3.94103,970.30 s
Fallback calls
0
Provider not reported
0

Answer checks

Answering and checking models.

These models answer and check the Guesser's questions and guesses. They are the same for every round and are not being tested.

Oracle

openai/gpt-5.6-luna

Searches for evidence and proposes an answer.

Calls
311
Cost
$0.7266
Reasoning
Medium
Routing
Fixed provider · openai
Resolved model
openai/gpt-5.6-luna
Resolved provider
OpenAI
Prompt version
live-web-oracle-v17-labelled-source-context
Provider details
OpenAI311 calls$0.72662,609.79 s
Fallback calls
0
Provider not reported
0
Reviewer

google/gemini-3.5-flash-lite

Checks each Oracle answer except Unknown, without seeing it.

Calls
308
Cost
$0.4126
Reasoning
Medium
Routing
Fixed provider · google-ai-studio
Resolved provider
Google AI Studio
Provider details
Google AI Studio308 calls$0.4126523.21 s
Fallback calls
0
Provider not reported
0
Judge

anthropic/claude-opus-5

Decides when the Oracle and Reviewer disagree.

Calls
12
Cost
$0.2585
Reasoning
Medium
Routing
Chosen automatically by OpenRouter
Resolved provider
Claude Platform on AWS
Provider details
Claude Platform on AWS12 calls$0.258569.53 s
Fallback calls
0
Provider not reported
0
Guess Validator

openai/gpt-5.6-luna

Checks whether a guess names the hidden subject.

Calls
32
Cost
$0.0062
Reasoning
Medium
Routing
Fixed provider · openai
Resolved model
openai/gpt-5.6-luna
Resolved provider
OpenAI
Prompt version
strict-guess-validator-v2-generic-kinds
Configuration
gpt-5.6-luna-validator
Provider details
OpenAI32 calls$0.006253.92 s
Fallback calls
0
Provider not reported
0