Official run
Claude Sonnet 5High
In this Twenty Questions LLM benchmark, Claude Sonnet 5 (high) ranks 10 of 18 with a 14.74 question score and 100% success. Lower is better. The run contains 35 scored episodes across 7 subjects with 5 trials per subject. Choose a subject to inspect its attempts.
Rank 10 of 18anthropicHigh
- Success
- 100%
- Contract compliance
- 97%
- Guesser cost
- $1.72
- Guesser time
- 49m 5s
Drill down
Subjects.
Complete benchmark
Run ledger.
All Guesser and benchmark-support activity.
- Total tokens
- 4,419,919
- Wall-clock runtime
- 2h 16m
- Guesser calls
- 551
- Completed
Reliability
Output contract breached.
16 invalid outputs affected 12 episodes and consumed 16 counted turns.
Provenance
- Execution
BX-20260728-official-M0005-012- Model ID
M-0005- Benchmark
B-0001- Base seed
0- Git commit
79fde3a043c3
Recorded cost
By role.
- Guesser34.4%$1.72
- Primary Oracle46.1%$2.31
- Reviewer9.3%$0.47
- Judge9.9%$0.49
- Validator0.3%$0.01
Roles in this cost:
- Guesser
- Asks the questions and submits the scored guess.
- Primary Oracle
- Searches for evidence and proposes an answer.
- Reviewer
- Checks each Oracle YES or NO independently.
- Judge
- Decides when the Oracle and Reviewer disagree.
- Validator
- Checks a submitted guess against the trusted subject.
Run configuration
Models.
Model identity, prompt versions, routing, calls, and recorded cost for the full run.
anthropic/claude-sonnet-5
Asks the questions and submits the scored guess.
- Calls
- 551
- Cost
- $1.7238
- Reasoning
- High
- Routing
- Exact provider · anthropic
- Resolved model
- anthropic/claude-sonnet-5
- Resolved provider
- Anthropic
- Prompt contract
stateful-category-guesser-v10-unknown-evidence-guidance- Configuration
M-0005
Resolved provider details
Legacy run. Per-call routing totals were not retained. The configured provider was anthropic.
Game support
Oracle and adjudication.
These models support the game. They are fixed across the run and are not under test.
openai/gpt-5.6-luna
Searches for evidence and proposes an answer.
- Calls
- 494
- Cost
- $2.3111
- Reasoning
- Medium
- Routing
- Exact provider · openai
- Resolved model
- openai/gpt-5.6-luna
- Resolved provider
- OpenAI
- Prompt contract
live-web-oracle-v7-direct-negative-evidence
Resolved provider details
Legacy run. Per-call routing totals were not retained. The configured provider was openai.
google/gemini-3.5-flash-lite
Checks each Oracle YES or NO independently.
- Calls
- 431
- Cost
- $0.4658
- Reasoning
- Medium
- Routing
- Exact provider · google-ai-studio
Resolved provider details
Legacy run. Per-call routing totals were not retained. The configured provider was google-ai-studio.
anthropic/claude-opus-5
Decides when the Oracle and Reviewer disagree.
- Calls
- 42
- Cost
- $0.4944
- Reasoning
- Medium
- Routing
- Exact provider · anthropic
Resolved provider details
Legacy run. Per-call routing totals were not retained. The configured provider was anthropic.
openai/gpt-5.6-luna
Checks a submitted guess against the trusted subject.
- Calls
- 41
- Cost
- $0.0132
- Reasoning
- Medium
- Routing
- Exact provider · openai
- Resolved model
- openai/gpt-5.6-luna
- Resolved provider
- OpenAI
- Prompt contract
strict-guess-validator-v1- Configuration
gpt-5.6-luna-validator
Resolved provider details
Legacy run. Per-call routing totals were not retained. The configured provider was openai.