Official run
Mistral Medium 3.5High
In this Twenty Questions AI benchmark, Mistral Medium 3.5 (high) ranks 17 of 18 with a question score of 20.06: the average number of questions per round, where lower is better. It failed 4 of 35 rounds. The run has 35 scored rounds: 7 subjects, 5 rounds each. Choose a subject to see its rounds.
Rank 17 of 18mistralHigh
Round scores are averaged for each subject, then across subjects. A failed round gets the fixed failure score.
The 95% confidence interval shows how certain the average is, based on repeated rounds on these subjects. A wider interval means less consistent results. It does not predict scores on other subjects.
- Solved
- 89%
- Valid format
- 99.7%
- Guesser cost
- $14.39
- Guesser time
- 8h 29m
Drill down
Subjects.
Complete benchmark
Run totals.
Includes the Guesser and all answer-checking models.
- Total tokens
- 7,228,806
- Total runtime
- 14h 58m
- Guesser calls
- 733
- Completed
Reply format
Format errors found.
2 invalid replies in 1 round used up 2 counted turns.
Run details
- Execution
BX-20260729-official-M0012-011- Model ID
M-0012- Benchmark
B-0001- Base seed
0- Git commit
e90f98eeaed0
Recorded cost
By role.
- Guesser77.6%$14.39
- Oracle16.8%$3.12
- Reviewer2.7%$0.50
- Judge2.5%$0.47
- Validator0.3%$0.06
Roles in this cost:
- Guesser
- The tested model. Asks questions and makes guesses.
- Oracle
- Searches for evidence and proposes an answer.
- Reviewer
- Checks each Oracle answer except Unknown, without seeing it.
- Judge
- Decides when the Oracle and Reviewer disagree.
- Validator
- Checks whether a guess names the hidden subject.
Run configuration
Models.
Models, prompt versions, providers, calls, and cost for the whole run.
mistralai/mistral-medium-3-5
The tested model. Asks questions and makes guesses.
- Calls
- 733
- Cost
- $14.3947
- Reasoning
- High
- Routing
- Fixed provider · mistral
- Resolved model
- mistralai/mistral-medium-3-5
- Resolved provider
- Mistral
- Prompt version
stateful-category-guesser-v10-unknown-evidence-guidance- Configuration
M-0012
Provider details
Older run: providers were not recorded per call. The configured provider was mistral.
Answer checks
Answering and checking models.
These models answer and check the Guesser's questions and guesses. They are the same for every round and are not being tested.
openai/gpt-5.6-luna
Searches for evidence and proposes an answer.
- Calls
- 545
- Cost
- $3.1247
- Reasoning
- Medium
- Routing
- Fixed provider · openai
- Resolved model
- openai/gpt-5.6-luna
- Resolved provider
- OpenAI
- Prompt version
live-web-oracle-v7-direct-negative-evidence
Provider details
Older run: providers were not recorded per call. The configured provider was openai.
google/gemini-3.5-flash-lite
Checks each Oracle answer except Unknown, without seeing it.
- Calls
- 508
- Cost
- $0.5018
- Reasoning
- Medium
- Routing
- Fixed provider · google-ai-studio
Provider details
Older run: providers were not recorded per call. The configured provider was google-ai-studio.
anthropic/claude-opus-5
Decides when the Oracle and Reviewer disagree.
- Calls
- 35
- Cost
- $0.4668
- Reasoning
- Medium
- Routing
- Fixed provider · anthropic
Provider details
Older run: providers were not recorded per call. The configured provider was anthropic.
openai/gpt-5.6-luna
Checks whether a guess names the hidden subject.
- Calls
- 186
- Cost
- $0.0606
- Reasoning
- Medium
- Routing
- Fixed provider · openai
- Resolved model
- openai/gpt-5.6-luna
- Resolved provider
- OpenAI
- Prompt version
strict-guess-validator-v1- Configuration
gpt-5.6-luna-validator
Provider details
Older run: providers were not recorded per call. The configured provider was openai.