Official run
DeepSeek V4.1 Flash 1017High
In this Twenty Questions AI benchmark, DeepSeek V4.1 Flash 1017 (high) ranks 21 of 23 with a question score of 22.70: the average number of questions per round, where lower is better. It failed 5 of 30 rounds. The run has 30 scored rounds: 10 subjects, 3 rounds each. Choose a subject to see its rounds.
Rank 21 of 23deepinfraHigh
Round scores are averaged for each subject, then across subjects. A failed round gets the fixed failure score.
The 95% confidence interval shows how certain the average is, based on repeated rounds on these subjects. A wider interval means less consistent results. It does not predict scores on other subjects.
- Solved
- 83%
- Valid format
- 99.3%
- Guesser cost
- $0.30
- Guesser time
- 5h 10m
Drill down
Subjects.
Complete benchmark
Run totals.
Includes the Guesser and all answer-checking models.
- Total tokens
- 7,128,149
- Total runtime
- 6h 56m
- Guesser calls
- 706
- Completed
Reply format
Format errors found.
5 invalid replies in 5 rounds used up 5 counted turns.
Run details
- Execution
BX-20261028-B-0003-official-M0037-001- Model ID
M-0037- Benchmark
B-0003- Base seed
0- Git commit
77ce4c7c18c6
Recorded cost
By role.
- Guesser9%$0.30
- Oracle47.3%$1.59
- Reviewer22.6%$0.76
- Judge20.9%$0.70
- Validator0.2%$0.01
Roles in this cost:
- Guesser
- The tested model. Asks questions and makes guesses.
- Oracle
- Searches for evidence and proposes an answer.
- Reviewer
- Checks each Oracle answer except Unknown, without seeing it.
- Judge
- Decides when the Oracle and Reviewer disagree.
- Validator
- Checks whether a guess names the hidden subject.
Run configuration
Models.
Models, prompt versions, providers, calls, and cost for the whole run.
deepseek/deepseek-v4.1-flash-1017
The tested model. Asks questions and makes guesses.
- Calls
- 706
- Cost
- $0.3010
- Reasoning
- High
- Routing
- Fixed provider · deepinfra
- Resolved model
- deepseek/deepseek-v4.1-flash-1017
- Resolved provider
- DeepInfra
- Prompt version
stateful-category-guesser-v16-five-answer-category-guide- Configuration
M-0037
Provider details
- Fallback calls
- 0
- Provider not reported
- 0
Answer checks
Answering and checking models.
These models answer and check the Guesser's questions and guesses. They are the same for every round and are not being tested.
openai/gpt-5.6-luna
Searches for evidence and proposes an answer.
- Calls
- 577
- Cost
- $1.5909
- Reasoning
- Medium
- Routing
- Fixed provider · openai
- Resolved model
- openai/gpt-5.6-luna
- Resolved provider
- OpenAI
- Prompt version
live-web-oracle-v17-labelled-source-context
Provider details
- Fallback calls
- 0
- Provider not reported
- 0
google/gemini-3.5-flash-lite
Checks each Oracle answer except Unknown, without seeing it.
- Calls
- 560
- Cost
- $0.7588
- Reasoning
- Medium
- Routing
- Fixed provider · google-ai-studio
- Resolved provider
- Google AI Studio
Provider details
- Fallback calls
- 0
- Provider not reported
- 0
anthropic/claude-opus-5
Decides when the Oracle and Reviewer disagree.
- Calls
- 32
- Cost
- $0.7041
- Reasoning
- Medium
- Routing
- Chosen automatically by OpenRouter
- Resolved provider
- Claude Platform on AWS
Provider details
- Fallback calls
- 0
- Provider not reported
- 0
openai/gpt-5.6-luna
Checks whether a guess names the hidden subject.
- Calls
- 36
- Cost
- $0.0079
- Reasoning
- Medium
- Routing
- Fixed provider · openai
- Resolved model
- openai/gpt-5.6-luna
- Resolved provider
- OpenAI
- Prompt version
strict-guess-validator-v2-generic-kinds- Configuration
gpt-5.6-luna-validator
Provider details
- Fallback calls
- 0
- Provider not reported
- 0