Official run

GLM-5.3-FlashHigh

In this Twenty Questions LLM benchmark, GLM-5.3-Flash (high) ranks 10 of 11 with a 23.27 question score and 73% success. Lower is better. The run contains 30 scored episodes across 10 subjects with 3 trials per subject. Choose a subject to inspect its attempts.

Rank 10 of 11z-aiHigh

Question score23.27questions · lower is better95% CI 19.58–26.96

Success: 73%
Contract compliance: 93%
Guesser cost: $0.05
Guesser time: 51m 31s

Drill down

Subjects.

01Albert Einstein10.00 avg 02Albert Schweitzer32.67 avg 03Garfield11.33 avg 04Achilles14.33 avg 05Genghis Khan29.33 avg 06Bike pump41.00 avg 07Spider web35.00 avg 08Eyebrow21.00 avg 09Moon10.33 avg 10Door handle27.67 avg

Complete benchmark

Run ledger.

All Guesser and benchmark-support activity.

Total tokens: 6,170,001
Wall-clock runtime: 2h 58m
Guesser calls: 720
Completed: Sep 8, 2026, 9:41:07 PM

Reliability

Output contract breached.

49 invalid outputs affected 30 episodes and consumed 49 counted turns.

Provenance

Execution: BX-20260908-B-0003-experimental-M0023-001
Model ID: M-0023
Benchmark: B-0003
Base seed: 0
Git commit: eceee31eef2e

Recorded cost

By role.

Cost composition.Share of full-run cost

Guesser1.6%$0.05
Primary Oracle53.8%$1.55
Reviewer24.1%$0.69
Judge20.1%$0.58
Validator0.3%$0.01

Excluded repair overhead$0.022 superseded attempts across 2 affected trials. Not included above.

Roles in this cost:

Guesser: Asks the questions and submits the scored guess.
Primary Oracle: Searches for evidence and proposes an answer.
Reviewer: Checks each Oracle YES or NO independently.
Judge: Decides when the Oracle and Reviewer disagree.
Validator: Checks a submitted guess against the trusted subject.

Read the role and answer-checking method →

Run configuration

Models.

Model identity, prompt versions, routing, calls, and recorded cost for the full run.

GuesserModel under test

z-ai/glm-5.3-flash

Asks the questions and submits the scored guess.

Calls: 720
Cost: $0.0471
Reasoning: High
Routing: Exact provider · z-ai
Resolved model: z-ai/glm-5.3-flash
Resolved provider: Z.AI
Prompt contract: stateful-category-guesser-v16-five-answer-category-guide
Configuration: M-0023

Resolved provider details

Z.AI720 calls$0.04713,090.80 s

Fallback calls: 0
Provider unreported: 0

Game support

Oracle and adjudication.

These models support the game. They are fixed across the run and are not under test.

Primary Oracle

openai/gpt-5.6-luna

Searches for evidence and proposes an answer.

Calls: 523
Cost: $1.5485
Reasoning: Medium
Routing: Exact provider · openai
Resolved model: openai/gpt-5.6-luna
Resolved provider: OpenAI
Prompt contract: live-web-oracle-v17-labelled-source-context

Resolved provider details

OpenAI523 calls$1.54855,373.07 s

Fallback calls: 0
Provider unreported: 0

Reviewer

google/gemini-3.5-flash-lite

Checks each Oracle YES or NO independently.

Calls: 501
Cost: $0.6949
Reasoning: Medium
Routing: Exact provider · google-ai-studio
Resolved provider: Google AI Studio

Resolved provider details

Google AI Studio501 calls$0.6949884.02 s

Fallback calls: 0
Provider unreported: 0

Judge

anthropic/claude-opus-5

Decides when the Oracle and Reviewer disagree.

Calls: 30
Cost: $0.5788
Reasoning: Medium
Routing: OpenRouter automatic routing
Resolved provider: Claude Platform on AWS

Resolved provider details

Claude Platform on AWS30 calls$0.5788172.98 s

Fallback calls: 0
Provider unreported: 0

Guess Validator

openai/gpt-5.6-luna

Checks a submitted guess against the trusted subject.

Calls: 45
Cost: $0.0089
Reasoning: Medium
Routing: Exact provider · openai
Resolved model: openai/gpt-5.6-luna
Resolved provider: OpenAI
Prompt contract: strict-guess-validator-v2-generic-kinds
Configuration: gpt-5.6-luna-validator

Resolved provider details

OpenAI45 calls$0.008978.04 s

Fallback calls: 0
Provider unreported: 0