Deep20Bench
v1.1Current

Benchmark edition

Edition 1.1 settings

10 subjects3 rounds per subject40 question limit
Question answersYesRather yesRather noNoUnknown
What changed
Source
Menu
Round

Round 2

Round 2

Identified Spider web in 17 counted questions.

Back to all Spider web rounds
Questions
17
Outcome
Solved
Format errors
Errors found View one example
Duration
17m 20s
Round cost
$0.0723 All 18 turns

Format warning

The model sent replies in the wrong format.

The Guesser sent 1 invalid reply. Each one before the question limit used up a counted turn. The model was only reminded of the required format; it was never told whether its question or guess was correct.

Valid format
94.4%
Format errors
1
Turns used by format errors
1

Transcript

Questions and answers.

Turns are shown in the order they happened. Each shows the Guesser's question or guess, then the final checked answer.

Exact Guesser setupSystem message and BEGIN event

The Guesser received these messages at the start of the round. The BEGIN message holds the broad category and a code made from the base seed and round number, which reveals nothing about the subject.

System message
You are the Guesser in a Twenty Questions benchmark. Identify the hidden subject using as few
counted turns as possible. BEGIN provides only its broad category. The variation_token has no
clues; do not interpret, mention, or repeat it.

The category in BEGIN has a broad meaning:
- person: a real human, living or historical.
- fictional_character: an invented character from a story, book, comic, film, television,
  game, or another fictional work.
- mythological_figure: a figure from mythology or traditional legend.
- video_game_character: a character known from video games.
- thing: a broad category covering natural and human-made entities, living or nonliving,
  whole entities or parts of them. Possibilities include everyday objects, tools, machines,
  materials, food, animals, plants, body parts, natural structures, geographical features,
  celestial bodies, natural phenomena, and concepts. These possibilities are not exhaustive.
  The target may be a general kind or one particular instance.

Use questions to narrow the supplied category. These definitions are not a list of targets.

Use ASK for a factual question that distinguishes candidates. Never use ASK to confirm a
specific identity. Use GUESS with a name and short identifying description to name a candidate.

ASK has five possible replies:
- YES or NO: the evidence establishes that answer to the exact question.
- RATHER_YES or RATHER_NO: evidence leans that way, but is not strong enough for a firm answer.
  Treat this as a clue; keep candidates fitting the opposite answer possible.
- UNKNOWN: no reliable direction. It supports neither answer.

Ask clear questions about verifiable properties. "Can", "sometimes", "usually", "always", and
"mainly" ask different things. Do not assume that one property proves another or rules out alternatives.
Qualified replies express uncertainty about your question, not how often a property applies.
Before narrowing heavily, check your key assumption with a different property. After rejected
guesses, reconsider earlier assumptions as well as candidates. Answers can be mistaken; check
another property when stuck. Do not ignore an answer merely because it conflicts with a
preferred candidate.

Each ASK, rejected GUESS, or format error before the limit costs one turn. A correct GUESS costs
none. After 40 counted turns, make one final GUESS; ASK is then forbidden. Identity
validation uses only YES, NO, or UNKNOWN. UNKNOWN on GUESS ends the game without success.

FORMAT_ERROR means your output broke the contract, was not checked for correctness, and cost
one turn. Retry using its displayed formats if a turn remains. Return only the required JSON
object with exactly one result action. No explanations, reasoning, or extra fields.
BEGIN event
{"category":"thing","event":"BEGIN","variation_token":"BKXG3AZI"}

Valid replies are shown in the required format. Rejected replies are shown separately, without provider or call IDs.

Q1
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Output limit exceeded · fixed format reminder sent

Why this reply was rejectedOutput limit exceeded · turn 1

The reply hit the output length limit before it was complete.

Recorded type
output_limit_exceeded
Turn
1
Counted
Yes · question 1
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider
Attempt 1 · finish length
{"result":{"action":"ASK","question":"Is the hidden subject a physical object or entity, as opposed to a purely abstract concept, event, or process?"
What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q2
1 · Guesser asks

Is the target a physical entity that exists in the material world?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target a physical entity that exists in the material world?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q3
1 · Guesser asks

Is the target human-made rather than occurring naturally?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target human-made rather than occurring naturally?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

In nature, spiders use their silk for several applications such as for webs, wrapping of prey, protection of their offspring and as a lifeline which ensures their safe escape from predators.

Open pmc.ncbi.nlm.nih.gov
Source 02

Source summary (model-reported)

Not all spiders build webs to catch their, largely insect, prey but those that do are responsible for some of the most iconic and beautiful structures in the living world. These three-dimensional criss-crosses of silk are built by ... spiders.

Open britishspiders.org.uk
2 · Answer check returns No
Q4
1 · Guesser asks

Is the target a living organism or a part of a living organism?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target a living organism or a part of a living organism?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The spiders generated sheet-webs from stiff, anchored threads and bundles of fine threads crossing each other, to which a hydrogel was added in several places. ... This is achieved by the use of a special web which is formed into a diving bell, the spider’s centre for rest and reproduction.

Open pmc.ncbi.nlm.nih.gov
2 · Answer check returns No
Q5
1 · Guesser asks

Is the target a celestial body or astronomical object (such as a planet, moon, star, or galaxy)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target a celestial body or astronomical object (such as a planet, moon, star, or galaxy)?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

Making a web is one of the many uses to which the spider puts silk to use. Spiders use silk for many types of webs in which food is ensnared, as well as for other structures and purposes.

Open irfi.org
2 · Answer check returns No
Q6
1 · Guesser asks

Is the target a geographical feature such as a mountain, river, lake, desert, or ocean?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target a geographical feature such as a mountain, river, lake, desert, or ocean?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q7
1 · Guesser asks

Is the target a natural substance or material (for example, a mineral, metal, or chemical compound)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target a natural substance or material (for example, a mineral, metal, or chemical compound)?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01
Spider silk refers to the material that spiders produce to make webs. The web is the architecture built from it, much as steel is a material and a bridge is a structure. Spider webs are often made with multiple types of silk.
Open theconversation.com
2 · Answer check returns No
Q8
1 · Guesser asks

Is the target a discrete, bounded object that a person could typically pick up and hold in their hand?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target a discrete, bounded object that a person could typically pick up and hold in their hand?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

A spider will start its web with extra-strong ‘structural’ silk to create the basic framework and anchor the web firmly to its surroundings with blobs of a tough, cement-like silk.

Open sussexwildlifetrust.org.uk
Source 02

Source summary (model-reported)

The primary function of a spider web is to catch prey... However, orb webs are fragile structures and often become damaged, leading to the need to either repair or rebuild the web.

Open sciencedirect.com
2 · Answer check returns No
Q9
1 · Guesser asks

Is the target a natural phenomenon or event, such as a weather event, atmospheric effect, or geological activity?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target a natural phenomenon or event, such as a weather event, atmospheric effect, or geological activity?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The spider's web is a structure created by a highly specific behavior pattern. The structure is a trap and thus its construction is an aspect of foraging. Spider webs are structures made from silk produced by spiders.

Open sciencedirect.com
Source 02

Source summary (model-reported)

The web acts like an air filter, trapping weak-flying insects... Most orb webs are rebuilt every day... To begin orb-web construction, the spider releases a silk thread that is carried by wind.

Open britannica.com
2 · Answer check returns No
Q10
1 · Guesser asks

Is the target too small to be seen with the naked eye (i.e., microscopic)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target too small to be seen with the naked eye (i.e., microscopic)?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q11
1 · Guesser asks

Is the target made primarily of rock or stone?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target made primarily of rock or stone?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The source states that orb webs consist of several different types of silk and that spider silk primarily consists of proteins containing amino acids such as glycine and alanine.

Open pmc.ncbi.nlm.nih.gov
Source 02

Source summary (model-reported)

The source describes spider orb webs as biological structures made from spider silk and characterizes spider silk as a unique protein-based material produced by spiders.

Open nature.com
Source 03

Source summary (model-reported)

The source states that spider silk is a protein fiber made from amino acids and that spider silk is used in different compositions for different parts of webs.

Open mcgill.ca
2 · Answer check returns No
Q12
1 · Guesser asks

Is the target larger than an average adult human being?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target larger than an average adult human being?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

The Smithsonian reports that Nephila spiders make the largest orb webs, which often exceed 3 feet (1 meter) in diameter.

Open si.edu
Source 03

Source summary (model-reported)

For Caerostris darwini, most bridgelines across a midsized river were 10–14 meters, with some reaching 25 meters; the paper describes these as among the largest orb webs known.

Open pmc.ncbi.nlm.nih.gov
2 · Answer check returns Unknown
Q13
1 · Guesser asks

Is the target typically constructed or produced by a non-human animal (for example, a nest, web, dam, or hive)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target typically constructed or produced by a non-human animal (for example, a nest, web, dam, or hive)?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01
Spiders make their webs from silk, a natural fibre made of protein. ... Spiders then follow various patterns of activity to construct their webs, depending on what species it is.
Open nhm.ac.uk
Source 02
To begin orb-web construction, the spider releases a silk thread ... The spider then descends from the center of the bridge, securing a thread on the ground or on a twig.
Open britannica.com
Source 03
Not all spiders build webs to catch their ... prey but those that do are responsible for some of the most iconic and beautiful structures in the living world. ... The orb web is the structure ... it is made by four spider families in Britain.
Open britishspiders.org.uk
2 · Answer check returns Yes
Q14
1 · Guesser asks

Is the target built by an insect?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target built by an insect?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q15
1 · Guesser asks

Is the target built by a bird?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target built by a bird?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 03
One of the lesser known hazards of a bird’s life... is collision with spiders’ webs. And when a bird flies through a web, it’s the spider’s hard work that takes the hit.
Open birdnote.org
2 · Answer check returns No
Q16
1 · Guesser asks

Is the target built by a mammal?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target built by a mammal?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

The Smithsonian states that “Spiders use their silk for several purposes, including web-building,” and explains that the silk is produced in glands and spun into fiber by spinnerets on the spider’s abdomen.

Open smithsonianmag.com
Source 03

Source summary (model-reported)

The Burke Museum describes a true web as “a silk structure made to catch prey” and explains that only some spider species catch prey by means of webs; it distinguishes webs from other silk structures made by spiders.

Open burkemuseum.org
2 · Answer check returns No
Q17
1 · Guesser asks

Is the target built by an arachnid, such as a spider?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the target built by an arachnid, such as a spider?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

Spiders make their webs from silk, a natural fibre made of protein. The page also describes a spider spinning silk to make its web.

Open nhm.ac.uk
Source 03

Source summary (model-reported)

Web building involves drawing out silk from spinnerets and weaving strands into geometric traps. Spiders that spin webs construct them in various forms, including tube-, funnel-, sheet-, and orb-shaped webs.

Open britannica.com
2 · Answer check returns Yes
G
1 · Guesser guessesNot counted

Spider web

A silken structure spun by spiders to catch prey, typically found in corners, between plants, or across openings.

Validator rationale The proposal identifies a spider web as an assembled structure built from spider silk, matching the target kind and function without contradicting any restrictions.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"A silken structure spun by spiders to catch prey, typically found in corners, between plants, or across openings.","name":"Spider web","question":null}}
2 · Validator returns Yes