Deep20Bench
v1.0Previous

Benchmark edition

Edition 1.0 settings

7 subjects5 rounds per subject50 question limit
Question answersYesNoUnknown
What changed
Source
Menu
Round

Round 1

Round 1

Identified Garfield in 33 counted questions.

Back to all Garfield rounds
Questions
33
Outcome
Solved
Format errors
Errors found View one example
Duration
13m 49s
Round cost
$0.5871 All 34 turns

Format warning

The model sent replies in the wrong format.

The Guesser sent 3 invalid replies. Each one before the question limit used up a counted turn. The model was only reminded of the required format; it was never told whether its question or guess was correct.

Valid format
91.2%
Format errors
3
Turns used by format errors
3

Transcript

Questions and answers.

Turns are shown in the order they happened. Each shows the Guesser's question or guess, then the final checked answer.

Exact Guesser setupSystem message and BEGIN event

The Guesser received these messages at the start of the round. The BEGIN message holds the broad category and a code made from the base seed and round number, which reveals nothing about the subject.

System message
You are the Guesser being evaluated in a Twenty Questions benchmark.

One subject is hidden. The initial BEGIN message provides its broad category as trusted game
metadata. Identify the subject using as few counted questions as possible. ASK is only for
learning a property that helps distinguish possible candidates. Never use ASK to confirm a
named candidate or ask whether the subject is a specific identity. When you are ready to name
a candidate, use GUESS immediately. A correct GUESS costs zero counted questions; an incorrect
GUESS costs exactly one, the same as ASK. A GUESS must provide both a name and a short
identifying description. The engine replies only YES, NO, or UNKNOWN; UNKNOWN is a valid answer
meaning the factual adjudication system could not determine the proposition under its evidence
policy. Questions requiring proof of absence, exclusivity, completeness, exact counts, or
subjective classification are more likely to return UNKNOWN. Prefer direct, positively
verifiable properties when possible.

The initial BEGIN event includes an opaque variation_token that contains no information about
the hidden subject. Do not interpret, repeat, mention, or derive meaning from it. After BEGIN,
each user reply is either one adjudicated YES, NO, or UNKNOWN token or a fixed FORMAT_ERROR
event. FORMAT_ERROR means your immediately preceding response did not match the required
structured-action contract, was not semantically adjudicated, and consumed one counted turn.
Use the formats in that event and try again. Never infer that the attempted question or
guess was right or wrong.

At most 50 questions or rejected guesses are counted. After that limit you receive
one final guess-only opportunity. Format errors also consume counted turns. The provider wire
response is an object with exactly one `result` action. On the final opportunity ASK is
forbidden. Return only the required structured response. Do not include analysis, hidden
reasoning, or additional fields.
BEGIN event
{"category":"fictional_character","event":"BEGIN","variation_token":"EAQCORIU"}

Valid replies are shown in the required format. Rejected replies are shown separately, without provider or call IDs.

Q1
1 · Guesser asks

Is the character from a book (originally, not primarily a movie or TV show)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the character from a book (originally, not primarily a movie or TV show)?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q2
1 · Guesser asks

Is the character from a movie (originally, not primarily a TV show, video game, or comic)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the character from a movie (originally, not primarily a TV show, video game, or comic)?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q3
1 · Guesser asks

Is the character from a video game?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the character from a video game?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02
In 1978 he launched Garfield ... Garfield has been represented in books, films, television specials, video games, amusement park attractions, and merchandise.
Open si.edu
2 · Answer check returns No
Q4
1 · Guesser asks

Is the character from a TV show (live-action or animated)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the character from a TV show (live-action or animated)?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q5
1 · Guesser asks

Is the character from a comic book or graphic novel?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the character from a comic book or graphic novel?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q6
1 · Guesser asks

Does this character originate from mythology or folklore (rather than a modern created work)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does this character originate from mythology or folklore (rather than a modern created work)?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q7
1 · Guesser asks

n

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"n"}}
2 · Answer check returns Unknown
Q8
1 · Guesser asks

Is the character from a stage play or musical?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the character from a stage play or musical?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q9
1 · Guesser asks

Is this character a mascot or spokes-character used in advertising or branding (e.g., for a product or company)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a mascot or spokes-character used in advertising or branding (e.g., for a product or company)?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q10
1 · Guesser asks

Is this character used to advertise a food or beverage product?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character used to advertise a food or beverage product?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q11
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Invalid action · fixed format reminder sent

Why this reply was rejectedInvalid action · turn 11

The reply was JSON, but it matched neither the ASK nor the GUESS format.

Recorded type
invalid_action
Turn
11
Counted
Yes · question 11
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider
Attempt 1 · finish stop
{"result":{"action":"ASK","question": "","name":null,"description":null}}
What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q12
1 · Guesser asks

Is this character used to advertise a cereal brand?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character used to advertise a cereal brand?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q13
1 · Guesser asks

Is this character an animal (rather than a human or humanoid character)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character an animal (rather than a human or humanoid character)?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q14
1 · Guesser asks

Is this character a tiger?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a tiger?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q15
1 · Guesser asks

Is this character a rabbit?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a rabbit?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q16
1 · Guesser asks

Is this character a bird?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a bird?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q17
1 · Guesser asks

}}{

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"}}{"}}
2 · Answer check returns Unknown
Q18
1 · Guesser asks

Is this character a bear?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a bear?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q19
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Invalid action · fixed format reminder sent

Why this reply was rejectedInvalid action · turn 19

The reply was JSON, but it matched neither the ASK nor the GUESS format.

Recorded type
invalid_action
Turn
19
Counted
Yes · question 19
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider
Attempt 1 · finish stop
{"result":{"action":"ASK","question":"","name":null,"description":null}}
What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q20
1 · Guesser asks

Is this character a monkey?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a monkey?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q21
1 · Guesser asks

Is this character a wolf or dog?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a wolf or dog?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q22
1 · Guesser asks

placeholder

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"placeholder"}}
2 · Answer check returns Unknown
Q23
1 · Guesser asks

Is this character a frog?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a frog?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q24
1 · Guesser asks

Is this character an insect (like a bee)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character an insect (like a bee)?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q25
1 · Guesser asks

Is this character a kangaroo?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a kangaroo?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q26
1 · Guesser asks

Is this character a cat?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this character a cat?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q27
1 · Guesser asks

Does this cat mascot appear on a cereal box for a chocolate or cocoa-flavored cereal?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does this cat mascot appear on a cereal box for a chocolate or cocoa-flavored cereal?"}}
2 · Answer check returns Unknown
Q28
1 · Guesser asks

Is this cat's fur primarily orange or orange-and-white colored?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this cat's fur primarily orange or orange-and-white colored?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q29
1 · Guesser asks

Does this cat character have a mostly humanoid/cartoon appearance (walks upright, wears clothes) rather than looking like a realistic cat?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does this cat character have a mostly humanoid/cartoon appearance (walks upright, wears clothes) rather than looking like a realistic cat?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q30
1 · Guesser asks

Is this cat mascot associated with a Kellogg's brand cereal (rather than General Mills, Post, or another company)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this cat mascot associated with a Kellogg's brand cereal (rather than General Mills, Post, or another company)?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q31
1 · Guesser asks

Is this cat mascot associated with a Post brand cereal?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this cat mascot associated with a Post brand cereal?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q32
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Invalid action · fixed format reminder sent

Why this reply was rejectedInvalid action · turn 32

The reply was JSON, but it matched neither the ASK nor the GUESS format.

Recorded type
invalid_action
Turn
32
Counted
Yes · question 32
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider
Attempt 1 · finish stop
{"result":{"action":"ASK","question":"","name":null,"description":null}}
What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q33
1 · Guesser asks

Is this cat mascot associated with a General Mills brand cereal?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is this cat mascot associated with a General Mills brand cereal?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
G
1 · Guesser guessesNot counted

Garfield

The famous orange and white cartoon cat known for laziness and love of lasagna, who has been used as a promotional mascot for various food products and cereals.

Validator rationale The proposed identity names Garfield and describes the same fictional orange, lazy, lasagna-loving cat character.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The famous orange and white cartoon cat known for laziness and love of lasagna, who has been used as a promotional mascot for various food products and cereals.","name":"Garfield","question":null}}
2 · Validator returns Yes