Deep20Bench
v1.1Current

Benchmark edition

Edition 1.1 settings

10 subjects3 rounds per subject40 question limit
Question answersYesRather yesRather noNoUnknown
What changed
Source
Menu
Round

Round 1

Round 1

Did not identify Eyebrow after 32 counted questions.

Back to all Eyebrow rounds
Questions
32 Scored as 41
Outcome
Too many format errors
Format errors
Errors found View one example
Duration
6m 44s
Round cost
$0.2621 All 32 turns

Format warning

The model sent replies in the wrong format.

The Guesser sent 7 invalid replies. Each one before the question limit used up a counted turn. The model was only reminded of the required format; it was never told whether its question or guess was correct.

Valid format
78.1%
Format errors
7
Turns used by format errors
7

Transcript

Questions and answers.

Turns are shown in the order they happened. Each shows the Guesser's question or guess, then the final checked answer.

Exact Guesser setupSystem message and BEGIN event

The Guesser received these messages at the start of the round. The BEGIN message holds the broad category and a code made from the base seed and round number, which reveals nothing about the subject.

System message
You are the Guesser in a Twenty Questions benchmark. Identify the hidden subject using as few
counted turns as possible. BEGIN provides only its broad category. The variation_token has no
clues; do not interpret, mention, or repeat it.

The category in BEGIN has a broad meaning:
- person: a real human, living or historical.
- fictional_character: an invented character from a story, book, comic, film, television,
  game, or another fictional work.
- mythological_figure: a figure from mythology or traditional legend.
- video_game_character: a character known from video games.
- thing: a broad category covering natural and human-made entities, living or nonliving,
  whole entities or parts of them. Possibilities include everyday objects, tools, machines,
  materials, food, animals, plants, body parts, natural structures, geographical features,
  celestial bodies, natural phenomena, and concepts. These possibilities are not exhaustive.
  The target may be a general kind or one particular instance.

Use questions to narrow the supplied category. These definitions are not a list of targets.

Use ASK for a factual question that distinguishes candidates. Never use ASK to confirm a
specific identity. Use GUESS with a name and short identifying description to name a candidate.

ASK has five possible replies:
- YES or NO: the evidence establishes that answer to the exact question.
- RATHER_YES or RATHER_NO: evidence leans that way, but is not strong enough for a firm answer.
  Treat this as a clue; keep candidates fitting the opposite answer possible.
- UNKNOWN: no reliable direction. It supports neither answer.

Ask clear questions about verifiable properties. "Can", "sometimes", "usually", "always", and
"mainly" ask different things. Do not assume that one property proves another or rules out alternatives.
Qualified replies express uncertainty about your question, not how often a property applies.
Before narrowing heavily, check your key assumption with a different property. After rejected
guesses, reconsider earlier assumptions as well as candidates. Answers can be mistaken; check
another property when stuck. Do not ignore an answer merely because it conflicts with a
preferred candidate.

Each ASK, rejected GUESS, or format error before the limit costs one turn. A correct GUESS costs
none. After 40 counted turns, make one final GUESS; ASK is then forbidden. Identity
validation uses only YES, NO, or UNKNOWN. UNKNOWN on GUESS ends the game without success.

FORMAT_ERROR means your output broke the contract, was not checked for correctness, and cost
one turn. Retry using its displayed formats if a turn remains. Return only the required JSON
object with exactly one result action. No explanations, reasoning, or extra fields.
BEGIN event
{"category":"thing","event":"BEGIN","variation_token":"EAQCORIU"}

Valid replies are shown in the required format. Rejected replies are shown separately, without provider or call IDs.

Q1
1 · Guesser asks

Is the subject a living organism?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject a living organism?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

Eyebrow hair serves many important biologic and aesthetic functions. This article reviews the structure and function of the hair follicle, as well as hair follicle morphogenesis and cycling.

Open pubmed.ncbi.nlm.nih.gov
2 · Answer check returns No
Q2
1 · Guesser asks

Was the subject created by humans?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was the subject created by humans?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The hair of your eyebrows is usually a bit thicker. All the hairs on your body grow out of tiny holes on the skin called follicles. ... It's in your genes ... [genes] determine if someone will grow a unibrow.

Open cas.wsu.edu
Source 02

Source summary (model-reported)

Eyebrows are an often overlooked feature of the face. ... Eyebrows act as a barrier, diverting moisture away from the eyes toward the sides of the face through their shape and the direction of their hairs.

Open britannica.com
2 · Answer check returns No
Q3
1 · Guesser asks

Is the subject primarily a solid object?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject primarily a solid object?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02
The eyebrows consist of serial arched rows (5–7 rows) of hair growing along the superior orbital margin. The eyebrows hair are short, thin but stiff, nonpliable with a slight curve along the body, and tapering toward the end.
Open pmc.ncbi.nlm.nih.gov
Source 03
Hair is a filamentous biomaterial consisting mainly of proteins in particular keratin. The structure of human hair is well known: the medulla is a loosely packed region near the centre of the hair surrounded by the cortex and cuticle.
Open pmc.ncbi.nlm.nih.gov
2 · Answer check returns Yes
Q4
1 · Guesser asks

Is the subject a celestial body such as a planet, moon, or asteroid?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject a celestial body such as a planet, moon, or asteroid?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

The eyebrow is described as an area of thick, short hairs above the eye and as one of the accessory visual structures that protect and support the eye.

Open en.wikipedia.org
2 · Answer check returns No
Q5
1 · Guesser asks

Is the subject a geological formation (such as a mountain, hill, canyon, or plateau)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject a geological formation (such as a mountain, hill, canyon, or plateau)?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q6
1 · Guesser asks

Is the subject a mineral?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject a mineral?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q7
1 · Guesser asks

Was the subject originally part of a living organism?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was the subject originally part of a living organism?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q8
1 · Guesser asks

Was the subject formed from fossilized tree resin?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was the subject formed from fossilized tree resin?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q9
1 · Guesser asks

Is the subject primarily composed of calcium carbonate?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject primarily composed of calcium carbonate?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

Human hair consists of 65–95 wt% proteins, 1–9 wt% lipids, 15–35 wt% water and less than 1% minerals and other trace components. Calcium is listed among trace elements.

Open sciencedirect.com
2 · Answer check returns No
Q10
1 · Guesser asks

Does the subject originate from an animal?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does the subject originate from an animal?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q11
1 · Guesser asks

Is the subject primarily composed of keratin?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject primarily composed of keratin?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q12
1 · Guesser asks

Is the subject typically found on mammals?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject typically found on mammals?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Rather yes
Q13
1 · Guesser guesses

Claw

A keratinous pointed digit used for grasping and defense, found on many animals including mammals.

Validator rationale The proposal identifies a claw, a different anatomical structure, rather than a human eyebrow.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"A keratinous pointed digit used for grasping and defense, found on many animals including mammals.","name":"Claw","question":null}}
2 · Validator returns No
Q14
1 · Guesser asks

Is the subject typically a protruding structure located on the head of an animal?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject typically a protruding structure located on the head of an animal?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The American Heritage Dictionary defines eyebrow as “the bony ridge extending over the eye” and “the arch of short hairs covering this ridge.” The Century Dictionary describes the brow as a “prominence” over the eye.

Open wordnik.com
Source 02

Source summary (model-reported)

Dictionary.com defines eyebrow as “the arch or ridge forming the upper part of the orbit of the eye” and also as “the fringe of hair growing on this arch or ridge.”

Open dictionary.com
2 · Answer check returns Yes
Q15
1 · Guesser asks

Does the subject have a bony core?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does the subject have a bony core?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02
The Eyebrows (supercilia) are two arched eminences of integument, which surmount the upper circumference of the orbits, and support numerous short, thick hairs, directed obliquely on the surface. The eyebrows consist of thickened integument, connected beneath with the Orbicularis oculi, Corrugator, and Frontalis muscles.
Open imaios.com
2 · Answer check returns No
Q16
1 · Guesser guesses

Rhinoceros horn

The solid keratinous horn of a rhinoceros, a protruding head structure without a bony core.

Validator rationale The target is a human eyebrow, while the proposal is a rhinoceros horn, a different anatomical structure.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The solid keratinous horn of a rhinoceros, a protruding head structure without a bony core.","name":"Rhinoceros horn","question":null}}
2 · Validator returns No
Q17
1 · Guesser asks

Does an individual animal typically have multiple instances of the subject?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does an individual animal typically have multiple instances of the subject?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

The eyebrow is defined as an arch of hair on the brow. The source discusses the eyebrow as a human periorbital anatomical feature and notes that eyebrow terminology may refer to a pair of structures.

Open elementsofmorphology.nih.gov
2 · Answer check returns Yes
Q18
1 · Guesser asks

Is the subject primarily used for sensory detection?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject primarily used for sensory detection?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

Britannica states that eyebrows act as a barrier diverting moisture, capturing small particles, and shielding the eyes from glare. It also describes their role in nonverbal communication and facial expressions.

Open britannica.com
Source 02

Source summary (model-reported)

Wikidoc identifies preventing moisture from reaching the eye as the main function of eyebrows. It additionally says eyebrows may provide a more sensitive sense for detecting objects near the eye.

Open wikidoc.org
2 · Answer check returns No
Q19
1 · Guesser guesses

Hair

A keratinous filament produced by mammals, commonly found on the head and other parts of the body

Validator rationale The target is the human eyebrow, an anatomical feature consisting of an area of short hairs. “Hair” identifies an individual filament or broad material/class, not the eyebrow as the whole body feature.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"A keratinous filament produced by mammals, commonly found on the head and other parts of the body","name":"Hair","question":null}}
2 · Validator returns No
Q20
1 · Guesser asks

Do the individual structures form overlapping plates that cover much of the animal’s body?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Do the individual structures form overlapping plates that cover much of the animal’s body?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

The eyebrows consist of serial arched rows (5–7 rows) of hair growing along the superior orbital margin. The eyebrows hair are short, thin but stiff, nonpliable with a slight curve along the body, and tapering toward the end.

Open pmc.ncbi.nlm.nih.gov
Source 03

Source summary (model-reported)

The area of curved, delicate hairs above our eyes are believed to channel unwanted moisture—sweat and rain in particular—away from the eye...

Open aao.org
2 · Answer check returns No
Q21
1 · Guesser asks

Do the structures have tiny barbs that easily detach and embed into a predator upon contact?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Do the structures have tiny barbs that easily detach and embed into a predator upon contact?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

The hair shaft is the visible, nongrowing portion of a hair protruding from the skin. The hair shaft has three layers: a central medulla, a keratinised cortex and an outer layer, known as the cuticle.

Open kenhub.com
Source 03

Source summary (model-reported)

The North American porcupine has about 30,000 quills, each one adorned with between 700 and 800 barbs along the 4 millimeters or so nearest its tip. The quills are readily shed and can become firmly embedded in an unfortunate victim.

Open science.org
2 · Answer check returns No
Q22
1 · Guesser asks

Is the subject primarily located on the lower part of the face, such as the chin area?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the subject primarily located on the lower part of the face, such as the chin area?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q23
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Output limit exceeded · fixed format reminder sent

Why this reply was rejectedOutput limit exceeded · turn 23

The reply hit the output length limit before it was complete.

Recorded type
output_limit_exceeded
Turn
23
Counted
Yes · question 23
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider

The provider returned no text.

What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q24
1 · Guesser guesses

Horn

A keratinous protruding head structure found on many mammals, typically appearing in pairs and lacking a bony core.

Validator rationale A horn is a different keratinous head structure, not the human eyebrow anatomical feature.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"A keratinous protruding head structure found on many mammals, typically appearing in pairs and lacking a bony core.","name":"Horn","question":null}}
2 · Validator returns No
Q25
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Output limit exceeded · fixed format reminder sent

Why this reply was rejectedOutput limit exceeded · turn 25

The reply hit the output length limit before it was complete.

Recorded type
output_limit_exceeded
Turn
25
Counted
Yes · question 25
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider

The provider returned no text.

What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q26
1 · Guesser guesses

Hedgehog spines

The sharp keratinous spines covering a hedgehog’s body, protruding from its head and used for defense.

Validator rationale The proposal identifies hedgehog spines, an animal defense structure, not the human eyebrow body feature.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The sharp keratinous spines covering a hedgehog’s body, protruding from its head and used for defense.","name":"Hedgehog spines","question":null}}
2 · Validator returns No
Q27
1 · Guesser guesses

Rhinoceros horns

The paired keratinous horns on a rhinoceros's head, composed of compacted keratin without a bony core.

Validator rationale Rhinoceros horns are a different anatomical feature from a human eyebrow.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The paired keratinous horns on a rhinoceros's head, composed of compacted keratin without a bony core.","name":"Rhinoceros horns","question":null}}
2 · Validator returns No
Q28
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Output limit exceeded · fixed format reminder sent

Why this reply was rejectedOutput limit exceeded · turn 28

The reply hit the output length limit before it was complete.

Recorded type
output_limit_exceeded
Turn
28
Counted
Yes · question 28
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider

The provider returned no text.

What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q29
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Output limit exceeded · fixed format reminder sent

Why this reply was rejectedOutput limit exceeded · turn 29

The reply hit the output length limit before it was complete.

Recorded type
output_limit_exceeded
Turn
29
Counted
Yes · question 29
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider

The provider returned no text.

What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q30
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Output limit exceeded · fixed format reminder sent

Why this reply was rejectedOutput limit exceeded · turn 30

The reply hit the output length limit before it was complete.

Recorded type
output_limit_exceeded
Turn
30
Counted
Yes · question 30
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider

The provider returned no text.

What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q31
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Output limit exceeded · fixed format reminder sent

Why this reply was rejectedOutput limit exceeded · turn 31

The reply hit the output length limit before it was complete.

Recorded type
output_limit_exceeded
Turn
31
Counted
Yes · question 31
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider

The provider returned no text.

What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q32
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Output limit exceeded · fixed format reminder sent

Why this reply was rejectedOutput limit exceeded · turn 32

The reply hit the output length limit before it was complete.

Recorded type
output_limit_exceeded
Turn
32
Counted
Yes · question 32
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider

The provider returned no text.

What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT