Deep20Bench
v1.0Previous

Benchmark edition

Edition 1.0 settings

7 subjects5 rounds per subject50 question limit
Question answersYesNoUnknown
What changed
Source
Menu
Round

Round 3

Round 3

Did not identify Albert Schweitzer after 50 counted questions.

Back to all Albert Schweitzer rounds
Questions
50 Scored as 51
Outcome
Too many format errors
Format errors
Errors found View one example
Duration
44m 36s
Round cost
$1.4264 All 51 turns

Format warning

The model sent replies in the wrong format.

The Guesser sent 1 invalid reply. Each one before the question limit used up a counted turn. The model was only reminded of the required format; it was never told whether its question or guess was correct.

Valid format
98.0%
Format errors
1
Turns used by format errors
1

Transcript

Questions and answers.

Turns are shown in the order they happened. Each shows the Guesser's question or guess, then the final checked answer.

Exact Guesser setupSystem message and BEGIN event

The Guesser received these messages at the start of the round. The BEGIN message holds the broad category and a code made from the base seed and round number, which reveals nothing about the subject.

System message
You are the Guesser being evaluated in a Twenty Questions benchmark.

One subject is hidden. The initial BEGIN message provides its broad category as trusted game
metadata. Identify the subject using as few counted questions as possible. ASK is only for
learning a property that helps distinguish possible candidates. Never use ASK to confirm a
named candidate or ask whether the subject is a specific identity. When you are ready to name
a candidate, use GUESS immediately. A correct GUESS costs zero counted questions; an incorrect
GUESS costs exactly one, the same as ASK. A GUESS must provide both a name and a short
identifying description. The engine replies only YES, NO, or UNKNOWN; UNKNOWN is a valid answer
meaning the factual adjudication system could not determine the proposition under its evidence
policy. Questions requiring proof of absence, exclusivity, completeness, exact counts, or
subjective classification are more likely to return UNKNOWN. Prefer direct, positively
verifiable properties when possible.

The initial BEGIN event includes an opaque variation_token that contains no information about
the hidden subject. Do not interpret, repeat, mention, or derive meaning from it. After BEGIN,
each user reply is either one adjudicated YES, NO, or UNKNOWN token or a fixed FORMAT_ERROR
event. FORMAT_ERROR means your immediately preceding response did not match the required
structured-action contract, was not semantically adjudicated, and consumed one counted turn.
Use the formats in that event and try again. Never infer that the attempted question or
guess was right or wrong.

At most 50 questions or rejected guesses are counted. After that limit you receive
one final guess-only opportunity. Format errors also consume counted turns. The provider wire
response is an object with exactly one `result` action. On the final opportunity ASK is
forbidden. Return only the required structured response. Do not include analysis, hidden
reasoning, or additional fields.
BEGIN event
{"category":"person","event":"BEGIN","variation_token":"AMZAISLM"}

Valid replies are shown in the required format. Rejected replies are shown separately, without provider or call IDs.

Q1
1 · Guesser asks

Is the person currently alive?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the person currently alive?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q2
1 · Guesser asks

Is the person male?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is the person male?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q3
1 · Guesser asks

Was the person born before 1900?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was the person born before 1900?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q4
1 · Guesser asks

Is he primarily known as a scientist, inventor, or mathematician?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he primarily known as a scientist, inventor, or mathematician?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q5
1 · Guesser asks

Was he primarily known as a political or military leader?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he primarily known as a political or military leader?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

Source 02
Profession: Doctor of Philosophy, Theology and Medicine. Musician and missionary. Motivation: Albert Schweitzer was nominated for his humanitarian work on a religious basis.
Open nobelprize.org
2 · Answer check returns No
Q6
1 · Guesser asks

Was he primarily known for work in the arts, such as writing, painting, music, or acting?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he primarily known for work in the arts, such as writing, painting, music, or acting?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q7
1 · Guesser asks

Was he primarily known as a philosopher or religious figure?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he primarily known as a philosopher or religious figure?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q8
1 · Guesser asks

Was he known as an explorer or adventurer?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he known as an explorer or adventurer?"}}
2 · Answer check returns Unknown
Q9
1 · Guesser asks

Was he known as an athlete or associated with sports?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he known as an athlete or associated with sports?"}}
2 · Answer check returns Unknown
Q10
1 · Guesser asks

Was he primarily known for criminal or notorious acts?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he primarily known for criminal or notorious acts?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q11
1 · Guesser asks

Was he primarily known for founding or running a major business or industry?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he primarily known for founding or running a major business or industry?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q12
1 · Guesser asks

Was he primarily known for contributions to medicine or psychology?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he primarily known for contributions to medicine or psychology?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q13
1 · Guesser asks

Was he primarily known for psychology or psychiatry rather than general medicine?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he primarily known for psychology or psychiatry rather than general medicine?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q14
1 · Guesser asks

Did he live mainly in the 19th century (born after 1750)?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Did he live mainly in the 19th century (born after 1750)?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q15
1 · Guesser asks

Was he born before the year 1 CE?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he born before the year 1 CE?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q16
1 · Guesser asks

Was he born before the year 1000 CE?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he born before the year 1000 CE?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q17
1 · Guesser asks

Was he European?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he European?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q18
1 · Guesser asks

Was he born after the year 1450?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he born after the year 1450?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q19
1 · Guesser asks

Is he best known for discovering or demonstrating the circulation of blood?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he best known for discovering or demonstrating the circulation of blood?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q20
1 · Guesser asks

Is he best known for pioneering modern human anatomy through dissection?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he best known for pioneering modern human anatomy through dissection?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q21
1 · Guesser asks

Is he best known for developing or popularizing vaccination?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he best known for developing or popularizing vaccination?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q22
1 · Guesser asks

Is he closely associated with alchemy or toxicology?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he closely associated with alchemy or toxicology?"}}
2 · Answer check returns Unknown
Q23
1 · Guesser asks

Was he primarily known as a surgeon or for advances in surgery?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he primarily known as a surgeon or for advances in surgery?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q24
1 · Guesser asks

Was he born before the year 1600?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he born before the year 1600?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q25
1 · Guesser asks

Is he best known for pioneering the use of the microscope to observe microorganisms or tissues?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he best known for pioneering the use of the microscope to observe microorganisms or tissues?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q26
1 · Guesser asks

Was he English?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he English?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q27
1 · Guesser guesses

Herman Boerhaave

Dutch physician and medical educator, a leading figure in 18th-century European clinical medicine.

Validator rationale The proposal identifies Herman Boerhaave, while the trusted subject is Albert Schweitzer.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"Dutch physician and medical educator, a leading figure in 18th-century European clinical medicine.","name":"Herman Boerhaave","question":null}}
2 · Validator returns No
Q28
1 · Guesser asks

Was he known for contributions to physiology or the study of digestion or metabolism?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he known for contributions to physiology or the study of digestion or metabolism?"}}
2 · Answer check returns Unknown
Q29
1 · Guesser asks

Was he Italian?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he Italian?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q30
1 · Guesser asks

Was he French?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he French?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q31
1 · Guesser asks

Is he best known as a pioneer of dentistry?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he best known as a pioneer of dentistry?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q32
1 · Guesser asks

Is a disease, medical sign, anatomical structure, or clinical term eponymously named after him?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is a disease, medical sign, anatomical structure, or clinical term eponymously named after him?"}}
2 · Answer check returns Unknown
Q33
1 · Guesser asks

Did he serve as a personal or chief physician to royalty or a royal court?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Did he serve as a personal or chief physician to royalty or a royal court?"}}
2 · Answer check returns Unknown
Q34
1 · Guesser guesses

Philippe Pinel

French physician (1745–1826) famed for pioneering the humane treatment of the mentally ill.

Validator rationale The proposed identity is Philippe Pinel, a different person from Albert Schweitzer.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"French physician (1745–1826) famed for pioneering the humane treatment of the mentally ill.","name":"Philippe Pinel","question":null}}
2 · Validator returns No
Q35
1 · Guesser asks

Is he credited with coining the term “orthopedics” or founding that medical specialty?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he credited with coining the term “orthopedics” or founding that medical specialty?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q36
1 · Guesser asks

Is he best known for inventing or popularizing the stethoscope and auscultation?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he best known for inventing or popularizing the stethoscope and auscultation?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q37
1 · Guesser asks

Is he noted for significant contributions to cardiology or the study of the heart?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he noted for significant contributions to cardiology or the study of the heart?"}}
2 · Answer check returns Unknown
Q38
1 · Guesser asks

Was he primarily associated with veterinary medicine or the care/study of animal health?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he primarily associated with veterinary medicine or the care/study of animal health?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q39
1 · Guesser asks

Was he born before the year 1700?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he born before the year 1700?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q40
1 · Guesser guesses

Félix Vicq d'Azyr

French physician and anatomist (1748–1794), founder of comparative anatomy and secretary of the Royal Society of Medicine.

Validator rationale The proposal identifies Félix Vicq d'Azyr, while the trusted subject is Albert Schweitzer.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"French physician and anatomist (1748–1794), founder of comparative anatomy and secretary of the Royal Society of Medicine.","name":"Félix Vicq d'Azyr","question":null}}
2 · Validator returns No
Q41
1 · Guesser asks

Was he associated with the Montpellier school of medicine or medical vitalism?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he associated with the Montpellier school of medicine or medical vitalism?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q42
1 · Guesser reply rejectedTurn counted

Reply in the wrong format.

Invalid action · fixed format reminder sent

Why this reply was rejectedInvalid action · turn 42

The reply was JSON, but it matched neither the ASK nor the GUESS format.

Recorded type
invalid_action
Turn
42
Counted
Yes · question 42
Feedback
FORMAT_ERROR
What the model returnedExact text from the provider
Attempt 1 · finish stop
{"result":{"action":"ASK","description":null,"name":null,"question":"Did he die before the year 1800?","question_canonical":null}}
What a valid reply looks likeThe same formats shown to the Guesser
ASK
{"result":{"action":"ASK","description":null,"name":null,"question":"<non-empty yes-or-no property question>"}}
GUESS
{"result":{"action":"GUESS","description":"<non-empty identifying description>","name":"<non-empty candidate name>","question":null}}

A reply must match one of these formats exactly. Text fields in use must not be empty; unused fields must be JSON null.

2 · System responseFORMAT
Q43
1 · Guesser asks

Did he die before the year 1800?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Did he die before the year 1800?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q44
1 · Guesser guesses

Paul-Joseph Barthez

French Montpellier-school physician and leading vitalist (1734–1806).

Validator rationale The proposal identifies Paul-Joseph Barthez, whereas the trusted subject is Albert Schweitzer.

Recorded Guesser outputRequired JSON format
{"result":{"action":"GUESS","description":"French Montpellier-school physician and leading vitalist (1734–1806).","name":"Paul-Joseph Barthez","question":null}}
2 · Validator returns No
Q45
1 · Guesser asks

Did he contribute medical articles to Diderot and d'Alembert's Encyclopédie?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Did he contribute medical articles to Diderot and d'Alembert's Encyclopédie?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q46
1 · Guesser asks

Is he best known as a founder of histology or for classifying bodily tissues?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he best known as a founder of histology or for classifying bodily tissues?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q47
1 · Guesser asks

Is he best known for work on hysteria, hypochondria, or the “vapors”?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Is he best known for work on hysteria, hypochondria, or the “vapors”?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No
Q48
1 · Guesser asks

Was he born after the year 1750?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he born after the year 1750?"}}
Oracle evidence1 model-reported source

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns Yes
Q49
1 · Guesser asks

Was he a professor at the University of Montpellier?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he a professor at the University of Montpellier?"}}
2 · Answer check returns Unknown
Q50
1 · Guesser asks

Was he known as one of the Idéologues?

Recorded Guesser outputRequired JSON format
{"result":{"action":"ASK","description":null,"name":null,"question":"Was he known as one of the Idéologues?"}}
Oracle evidence2 model-reported sources

The Oracle reported these excerpts and URLs. They are not independently checked.

2 · Answer check returns No