Official run

Gemini 3.7 FlashHigh

In this Twenty Questions LLM benchmark, Gemini 3.7 Flash (high) ranks 7 of 18 with a 14.03 question score and 97% success. Lower is better. The run contains 35 scored episodes across 7 subjects with 5 trials per subject. Choose a subject to inspect its attempts.

Rank 7 of 18google-ai-studioHigh

Question score14.03questions · lower is better95% CI 10.78–17.28

Success: 97%
Contract compliance: 100%
Guesser cost: $2.28
Guesser time: 44m 51s

Drill down

Subjects.

01Albert Einstein12.00 avg 02Albert Schweitzer22.40 avg 03Stephen King13.80 avg 04Garfield11.40 avg 05Achilles18.80 avg 06Genghis Khan11.60 avg 07Mario8.20 avg

Complete benchmark

Run ledger.

All Guesser and benchmark-support activity.

Total tokens: 6,970,713
Wall-clock runtime: 2h 40m
Guesser calls: 525
Completed: Aug 17, 2026, 1:58:15 PM

Reliability

Output contract clean.

All 525 evaluated outputs matched the public structured-action contract.

Provenance

Execution: BX-20260817-official-M0016-015
Model ID: M-0016
Benchmark: B-0001
Base seed: 0
Git commit: 7d637b2e6b71

Recorded cost

By role.

Cost composition.Share of full-run cost

Guesser23.8%$2.28
Primary Oracle62.3%$5.97
Reviewer5%$0.48
Judge8.9%$0.85
Validator0%$0.00

Roles in this cost:

Guesser: Asks the questions and submits the scored guess.
Primary Oracle: Searches for evidence and proposes an answer.
Reviewer: Checks each Oracle YES or NO independently.
Judge: Decides when the Oracle and Reviewer disagree.
Validator: Checks a submitted guess against the trusted subject.

Read the role and answer-checking method →

Run configuration

Models.

Model identity, prompt versions, routing, calls, and recorded cost for the full run.

GuesserModel under test

google/gemini-3.7-flash

Asks the questions and submits the scored guess.

Calls: 525
Cost: $2.2840
Reasoning: High
Routing: Exact provider · google-ai-studio
Resolved model: google/gemini-3.7-flash
Resolved provider: Google AI Studio
Prompt contract: stateful-category-guesser-v10-unknown-evidence-guidance
Configuration: M-0016

Resolved provider details

Google AI Studio525 calls$2.28402,690.96 s

Fallback calls: 0
Provider unreported: 0

Game support

Oracle and adjudication.

These models support the game. They are fixed across the run and are not under test.

Primary Oracle

openai/gpt-5.6-luna

Searches for evidence and proposes an answer.

Calls: 481
Cost: $5.9667
Reasoning: Medium
Routing: Exact provider · openai
Resolved model: openai/gpt-5.6-luna
Resolved provider: OpenAI
Prompt contract: live-web-oracle-v8-classified-research

Resolved provider details

OpenAI495 calls$5.96675,882.65 s

Fallback calls: 0
Provider unreported: 0

Reviewer

google/gemini-3.5-flash-lite

Checks each Oracle YES or NO independently.

Calls: 466
Cost: $0.4781
Reasoning: Medium
Routing: Exact provider · google-ai-studio
Resolved provider: Google AI Studio

Resolved provider details

Google AI Studio466 calls$0.4781650.76 s

Fallback calls: 0
Provider unreported: 0

Judge

anthropic/claude-opus-5

Decides when the Oracle and Reviewer disagree.

Calls: 67
Cost: $0.8481
Reasoning: Medium
Routing: OpenRouter automatic routing
Resolved provider: Amazon Bedrock

Resolved provider details

Amazon Bedrock67 calls$0.8481312.67 s

Fallback calls: 0
Provider unreported: 0

Guess Validator

openai/gpt-5.6-luna

Checks a submitted guess against the trusted subject.

Calls: 43
Cost: $0.0027
Reasoning: Medium
Routing: Exact provider · openai
Resolved model: openai/gpt-5.6-luna
Resolved provider: OpenAI
Prompt contract: strict-guess-validator-v1
Configuration: gpt-5.6-luna-validator

Resolved provider details

OpenAI43 calls$0.002764.47 s

Fallback calls: 0
Provider unreported: 0