Deep20Bench
v1.1Current

Benchmark edition

Edition 1.1 settings

10 subjects3 rounds per subject40 question limit
Question answersYesRather yesRather noNoUnknown
What changed
Source
Menu

Public data

Deep20Bench 1.1 dataset.

Deep20Bench publishes all the data behind its leaderboard, not only the averages: runs, rounds, transcripts, evidence, outcomes, cost, and timing. Project-authored data may be reused under CC BY 4.0 with attribution; third-party evidence excerpts are not covered.

  • Trace each published result.
  • Reproduce the official summaries.
  • Build your own evaluations.

01 Choose a format

Pick the file you need.

Start with the JSON file. It holds everything in the CSV file and more.

CSV · UTF-8

Leaderboard table

One row per ranked model. Long decimals are rounded to 6 places; the JSON file keeps exact values.

Download CSV (6 KB) ↓

JSON Schema · draft 2020-12

Public data schema

The structure, required fields, and allowed values of data format v10.

Download schema (61 KB) ↓

Older v9 download

The v9 dataset and v9 schema are still updated for edition 1.0. Version 9 cannot hold Rather yes and Rather no answers, so edition 1.1 results are only in the v10 files above.

02 Explore the file

Find the main records.

These four paths cover rankings, runs, rounds, and transcripts.

leaderboard[]
Ranks, models, question scores, solved rounds, format errors, cost, and time.
official_runs[]
Each ranked run in full: settings, totals, subjects, and all models used.
subjects[].trials[]
Each round's outcome, counted questions, score, cost, duration, and format errors.
trials[].episode
The round transcript: questions, answers, evidence, usage, timing, and format errors.

Example query

Print each model's rank, name, and exact question score.

jq -r '.leaderboard[] | [.rank, .model.display_name, .question_score] | @tsv' deep20bench-v10.json

CSV column notes

ideal_distance_rank
The rank shown on the Efficiency page.
efficiency_rank
An older cost-adjusted rank, the same as product_efficiency_rank. Kept for existing users of the file.
contract_status
Reply format: clean (no format errors), breached (at least one format error), or not_evaluable (not checked).

03 Verify and reuse

Check the release details.

See what went into this build, then cite and reuse it.

Run files checked
43 All editions, including replaced and unranked runs.
Ranked runs
20 The newest complete run of each model in this edition.
Game engine version
v9 Not the data format version.
Latest completion
Edition rules ID
qualified-core-v1Names the accepted settings of this edition in the data.
Subject catalog SHA-256
ff2fb52f3aa2213913e9b979211e55cc73b7bb9ae5185b44bb2a312dd1c4965a

Contents

What is and is not published.

Included

IDs, model settings, scores, outcomes, costs, timestamps, the source code commit, format errors, transcripts, Oracle evidence, and the source of each reused answer. Format error records can include the Guesser's invalid reply text.

Excluded

The answer-checking models' instructions, raw responses, and internal decisions; hidden reasoning; full provider messages; credentials; and private subject data. Private diagnostic files are not published.

The site is built from finished, signed run files. Building it makes no model calls and does not change the run files.

Data license

CC BY 4.0

Project-authored result data may be reused with attribution. Third-party evidence excerpts are excluded.

CC BY 4.0 data license ↗

Software license

Source available

The software is source-available under a dual-license model: PolyForm Noncommercial for noncommercial use, with separate commercial licenses available.

Software licensing - source-available ↗