Deep20Bench
v1.1Current

Benchmark edition

Edition 1.1 settings

10 subjects3 rounds per subject40 question limit
Question answersYesRather yesRather noNoUnknown
What changed
Source
Menu
Round

Round 2

Round 2

Did not identify Moon after 40 counted questions.

Back to all Moon rounds
Questions
40 Scored as 41
Outcome
Too many format errors
Format errors
Errors found View one example
Duration
4m 21s
Round cost
$0.0643 All 41 turns

Format warning

The model sent replies in the wrong format.

The Guesser sent 1 invalid reply. Each one before the question limit used up a counted turn. The model was only reminded of the required format; it was never told whether its question or guess was correct.

Valid format
97.6%
Format errors
1
Turns used by format errors
1

Transcript

Questions and answers.

Turns are shown in the order they happened. Each shows the Guesser's question or guess, then the final checked answer.

Exact Guesser setupSystem message and BEGIN event

The Guesser received these messages at the start of the round. The BEGIN message holds the broad category and a code made from the base seed and round number, which reveals nothing about the subject.

System message
You are the Guesser in a Twenty Questions benchmark. Identify the hidden subject using as few
counted turns as possible. BEGIN provides only its broad category. The variation_token has no
clues; do not interpret, mention, or repeat it.

The category in BEGIN has a broad meaning:
- person: a real human, living or historical.
- fictional_character: an invented character from a story, book, comic, film, television,
  game, or another fictional work.
- mythological_figure: a figure from mythology or traditional legend.
- video_game_character: a character known from video games.
- thing: a broad category covering natural and human-made entities, living or nonliving,
  whole entities or parts of them. Possibilities include everyday objects, tools, machines,
  materials, food, animals, plants, body parts, natural structures, geographical features,
  celestial bodies, natural phenomena, and concepts. These possibilities are not exhaustive.
  The target may be a general kind or one particular instance.

Use questions to narrow the supplied category. These definitions are not a list of targets.

Use ASK for a factual question that distinguishes candidates. Never use ASK to confirm a
specific identity. Use GUESS with a name and short identifying description to name a candidate.

ASK has five possible replies:
- YES or NO: the evidence establishes that answer to the exact question.
- RATHER_YES or RATHER_NO: evidence leans that way, but is not strong enough for a firm answer.
  Treat this as a clue; keep candidates fitting the opposite answer possible.
- UNKNOWN: no reliable direction. It supports neither answer.

Ask clear questions about verifiable properties. "Can", "sometimes", "usually", "always", and
"mainly" ask different things. Do not assume that one property proves another or rules out alternatives.
Qualified replies express uncertainty about your question, not how often a property applies.
Before narrowing heavily, check your key assumption with a different property. After rejected
guesses, reconsider earlier assumptions as well as candidates. Answers can be mistaken; check
another property when stuck. Do not ignore an answer merely because it conflicts with a
preferred candidate.

Each ASK, rejected GUESS, or format error before the limit costs one turn. A correct GUESS costs
none. After 40 counted turns, make one final GUESS; ASK is then forbidden. Identity
validation uses only YES, NO, or UNKNOWN. UNKNOWN on GUESS ends the game without success.

FORMAT_ERROR means your output broke the contract, was not checked for correctness, and cost
one turn. Retry using its displayed formats if a turn remains. Return only the required JSON
object with exactly one result action. No explanations, reasoning, or extra fields.
BEGIN event
{"category":"thing","event":"BEGIN","variation_token":"BKXG3AZI"}

Valid replies are shown in the required format. Rejected replies are shown separately, without provider or call IDs.

12 of 29 answers in this round reuse an earlier, fully checked answer with its original evidence. No new search or check ran for those turns.

Q1
1 · Guesser asks

Is it a man-made object?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it a man-made object?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The Moon, the natural satellite of Earth. The Moon is a rocky body that orbits planet Earth. It likely formed when a very large (Mars-sized) asteroid struck Earth 4.5 billion years ago.

Open asc-csa.gc.ca
2 · Answer check returns No
Q2
1 · Guesser asks

Is it a living organism?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it a living organism?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q3
1 · Guesser asks

Is it found on Earth?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it found on Earth?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

NASA identifies the Moon as Earth's satellite and provides educational material specifically about Earth's Moon.

Open nasa.gov
2 · Answer check returns No
Q4
1 · Guesser asks

Is it a celestial body?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it a celestial body?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q5
1 · Guesser asks

Is it a planet?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it a planet?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

Our moon is the fifth largest of the 190+ moons orbiting planets in our solar system. Earth's only natural satellite is simply called "the Moon".

Open arcnav.psi.edu
2 · Answer check returns No
Q6
1 · Guesser asks

Is it a star?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it a star?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The source distinguishes stars, planets, dwarf planets, asteroids, and moons. It states that stars produce their own energy through nuclear fusion, while moons are celestial bodies orbiting a planet or another small celestial body and do not generate their own light.

Open asc-csa.gc.ca
Source 02

Source summary (model-reported)

The Canadian Space Agency describes the Moon as Earth's natural satellite and gives its average distance from Earth, mass, orbital period, and other characteristics.

Open asc-csa.gc.ca
2 · Answer check returns No
Q7
1 · Guesser asks

Is it smaller than a planet?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it smaller than a planet?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The Moon is about one-quarter the size of Earth in diameter... Moon has 3474 km in diameter, while the Earth is 12 756 km.

Open esa.int
2 · Answer check returns Yes
Q8
1 · Guesser asks

Does it orbit a star?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does it orbit a star?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q9
1 · Guesser asks

Is it in our solar system?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it in our solar system?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

NASA identifies the subject as “The Moon: Earth’s Satellite” and categorizes the material under “Solar System and Planets, Earth's Moon.”

Open nasa.gov
2 · Answer check returns Yes
Q10
1 · Guesser asks

Is it larger than 1 km in diameter?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it larger than 1 km in diameter?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q11
1 · Guesser asks

Is it a dwarf planet?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it a dwarf planet?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01
A "dwarf planet" is a celestial body that (a) is in orbit around the Sun, (b) has sufficient mass for its self-gravity to overcome rigid body forces so that it assumes a hydrostatic equilibrium (nearly round) shape, (c) has not cleared the neighbourhood around its orbit, and (d) is not a satellite.
Open iau.org
2 · Answer check returns No
Q12
1 · Guesser asks

Is it larger than 1000 km in diameter?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it larger than 1000 km in diameter?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q13
1 · Guesser asks

Does it have a known atmosphere?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does it have a known atmosphere?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01
Does the Moon have an atmosphere? Yes, but just barely, and it’s very different from the atmosphere we know on Earth. ... The lunar atmosphere is mostly an exosphere, which itself is mostly empty space.
Open science.nasa.gov
2 · Answer check returns Yes
Q14
1 · Guesser asks

Does it have more than one known moon?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does it have more than one known moon?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q15
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Invalid action · fixed format reminder sent

Why this reply was rejectedInvalid action · turn 15

The reply was JSON, but it matched neither the ASK nor the GUESS format.

Recorded type
invalid_action
Turn
15
Counted
Yes · question 15
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider
Attempt 1 · finish stop
{ "result": { "action": "GUESS", "question": null, "name": "Pluto", "description": "A dwarf planet or large object in the outer solar system with a thin atmosphere and one main moon (Charon), but it was classified as not a dwarf planet. Wait, let me think again. The answer to dwarf planet was NO, and it has an atmosphere, and one known moon. Let me reconsider. Pluto is a dwarf planet, so that rules it out. Maybe Triton? It orbits Neptune, has an atmosphere, and is large (about 2710 km). It has no known moons of its own (or it does? No, Triton has no moons). Wait, Triton has no moons, so it wouldn't have more than one. Actually, Triton has no moons. Hmm. Let me think of other options. Titan is a moon, but it orbits Saturn, has an atmosphere, and has no moons of its own. Wait, Titan has no known moons. The question is about the target object itself. Does Titan have moons? No. So 'more than one known moon' - Titan has zero. Wait, the answer was NO to 'more than one known moon'. So it has 0 or 1 moon. Titan has 0 moons. Triton has 0 moons. But Triton is in our solar system, has atmosphere, is large (>1000 km), is not a dwarf planet. Let me guess Titan or Triton." } }
What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q16
1 · Guesser asks

Does it have a known atmosphere?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does it have a known atmosphere?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01
Does the Moon have an atmosphere? Yes, but just barely, and it’s very different from the atmosphere we know on Earth. ... The lunar atmosphere is mostly an exosphere, which itself is mostly empty space.
Open science.nasa.gov
2 · Answer check returns Yes
Q17
1 · Guesser asks

Does it have more than one known moon?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does it have more than one known moon?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q18
1 · Guesser guesses

Mars

The fourth planet from the Sun in our solar system

Validator rationale The proposal identifies Mars, a different celestial body, whereas the target is Earth's Moon.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The fourth planet from the Sun in our solar system","name":"Mars","question":null}}
2 · Validator returns No
Q19
1 · Guesser asks

Does it have a known atmosphere?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does it have a known atmosphere?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01
Does the Moon have an atmosphere? Yes, but just barely, and it’s very different from the atmosphere we know on Earth. ... The lunar atmosphere is mostly an exosphere, which itself is mostly empty space.
Open science.nasa.gov
2 · Answer check returns Yes
Q20
1 · Guesser asks

Is it the innermost of the planets with an atmosphere?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it the innermost of the planets with an atmosphere?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The terrestrial planets are the four innermost planets in the solar system: Mercury, Venus, Earth, and Mars. ... The planets Venus, Earth, and Mars have significant atmospheres, while Mercury has almost none.

Open science.nasa.gov
Source 02

Source summary (model-reported)

Moons are natural objects that orbit planets. Scientists usually refer to them as planetary satellites ... Earth has one [moon].

Open esa.int
2 · Answer check returns No
Q21
1 · Guesser asks

Is it larger than Earth?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it larger than Earth?"}}
Oracle evidence3 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The Moon is about one-quarter the size of Earth in diameter... Moon has 3474 km in diameter, while the Earth is 12 756 km.

Open esa.int
Source 03

Source summary (model-reported)

The Earth is approximately 7,900 miles (12,800 kilometers) in diameter, or about four times the diameter of the Moon, 2,100 miles (3,500 kilometers).

Open science.nasa.gov
2 · Answer check returns No
Q22
1 · Guesser guesses

Venus

The second planet from the Sun in our solar system

Validator rationale The target is Earth's Moon, while the proposal identifies Venus, a different celestial body.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The second planet from the Sun in our solar system","name":"Venus","question":null}}
2 · Validator returns No
Q23
1 · Guesser asks

Is it the outermost planet in our solar system?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it the outermost planet in our solar system?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q24
1 · Guesser asks

Is it the innermost of the planets with an atmosphere?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it the innermost of the planets with an atmosphere?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

The terrestrial planets are the four innermost planets in the solar system: Mercury, Venus, Earth, and Mars. ... The planets Venus, Earth, and Mars have significant atmospheres, while Mercury has almost none.

Open science.nasa.gov
Source 02

Source summary (model-reported)

Moons are natural objects that orbit planets. Scientists usually refer to them as planetary satellites ... Earth has one [moon].

Open esa.int
2 · Answer check returns No
Q25
1 · Guesser asks

Is it the outermost planet with an atmosphere?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it the outermost planet with an atmosphere?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q26
1 · Guesser asks

Is it the second outermost planet in our solar system?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it the second outermost planet in our solar system?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

Our solar system has eight planets: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. ... Uranus is the seventh planet from the Sun ... Neptune is the eighth and most distant planet in our solar system.

Open solarsystem.nasa.gov
Source 02

Source summary (model-reported)

Our solar system consists of our star, the Sun, and everything bound to it by gravity – the planets Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune; dwarf planets such as Pluto; dozens of moons; and millions of asteroids, comets, and meteoroids.

Open solarsystem.nasa.gov
2 · Answer check returns No
Q27
1 · Guesser asks

Is it smaller than Earth?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it smaller than Earth?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01

Source summary (model-reported)

NASA's comparison states: "Earth's Moon is 3.7x smaller than Earth." It also lists equatorial radius as 1,737.5 km for the Moon and 6,371.00 km for Earth.

Open science.nasa.gov
2 · Answer check returns Yes
Q28
1 · Guesser guesses

Mercury

The innermost planet from the Sun in our solar system

Validator rationale The target is Earth's Moon, while the proposal identifies Mercury, a different celestial body.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The innermost planet from the Sun in our solar system","name":"Mercury","question":null}}
2 · Validator returns No
Q29
1 · Guesser guesses

Titan

The largest moon of Saturn

Validator rationale Titan is a different specific celestial body—a moon of Saturn—not Earth's Moon.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The largest moon of Saturn","name":"Titan","question":null}}
2 · Validator returns No
Q30
1 · Guesser asks

Is it larger than Titan?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it larger than Titan?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01
Titan is bigger than Earth's moon... Titan has a radius of about 1,600 miles (2,575 kilometers), and is nearly 50 percent wider than Earth’s moon... Diameter 3,200 miles (5,149.4 kilometers).
Open solarsystem.nasa.gov
2 · Answer check returns No
Q31
1 · Guesser guesses

Ganymede

The largest moon in the solar system, orbiting Jupiter

Validator rationale The target is Earth's Moon, while Ganymede is a different natural satellite orbiting Jupiter.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The largest moon in the solar system, orbiting Jupiter","name":"Ganymede","question":null}}
2 · Validator returns No
Q32
1 · Guesser asks

Does it orbit the largest planet in our solar system?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does it orbit the largest planet in our solar system?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q33
1 · Guesser guesses

Pluto

A dwarf planet in the Kuiper belt, formerly considered the ninth planet

Validator rationale The target is Earth's Moon, a specific celestial body; Pluto is a different celestial body.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"A dwarf planet in the Kuiper belt, formerly considered the ninth planet","name":"Pluto","question":null}}
2 · Validator returns No
Q34
1 · Guesser asks

Is it a planet?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it a planet?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02

Source summary (model-reported)

Our moon is the fifth largest of the 190+ moons orbiting planets in our solar system. Earth's only natural satellite is simply called "the Moon".

Open arcnav.psi.edu
2 · Answer check returns No
Q35
1 · Guesser asks

Is it larger than 1000 km?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is it larger than 1000 km?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q36
1 · Guesser guesses

Charon

The largest moon of Pluto

Validator rationale Charon is a different specific natural satellite, whereas the target is Earth's Moon.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The largest moon of Pluto","name":"Charon","question":null}}
2 · Validator returns No
Q37
1 · Guesser guesses

Triton

The largest moon of Neptune

Validator rationale Triton is Neptune's moon, a different specific celestial body from Earth's Moon.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The largest moon of Neptune","name":"Triton","question":null}}
2 · Validator returns No
Q38
1 · Guesser guesses

Ceres

The smallest dwarf planet in the asteroid belt, also the largest object in the asteroid belt

Validator rationale Ceres is a different specific celestial body from the Moon.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"The smallest dwarf planet in the asteroid belt, also the largest object in the asteroid belt","name":"Ceres","question":null}}
2 · Validator returns No
Q39
1 · Guesser asks

Does it have a known atmosphere?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Does it have a known atmosphere?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 01
Does the Moon have an atmosphere? Yes, but just barely, and it’s very different from the atmosphere we know on Earth. ... The lunar atmosphere is mostly an exosphere, which itself is mostly empty space.
Open science.nasa.gov
2 · Answer check returns Yes
Q40
1 · Guesser guesses

Io

A large moon of Jupiter

Validator rationale Io is a specific moon of Jupiter, whereas the target is Earth's Moon.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"A large moon of Jupiter","name":"Io","question":null}}
2 · Validator returns No