Deep20Bench
v.1.1Current

Benchmark edition

Edition 1.1 settings

10 subjects3 rounds per subject40 question limit
Question answersYESRather yesRather noNOUNKNOWN
What changed
Source
Menu

Public data

Edition 1.1 data.

Deep20Bench publishes the data behind its leaderboard, not only aggregate scores: runs, trials, transcripts, evidence, outcomes, cost, and timing.

Project-authored result data and documentation use CC BY 4.0 and may be reused with attribution.

  • Trace each published result.
  • Reproduce the official summaries.
  • Build your own evaluations.

01 Choose a format

Start with the file that fits your task.

JSON carries the complete public record, CSV carries the current rankings, and the schema validates the JSON dataset.

JSON · schema v10

Full public dataset

Cohort rules, models, runs, subjects, episodes, scores, and build details.

Download JSON ↓

CSV · UTF-8

Leaderboard table

One row per model with a selected complete run.

Download CSV ↓

JSON Schema · draft 2020-12

Public data schema

Types, required fields, enums, and nested public objects for schema v10.

Download schema ↓

Existing v9 download

The v9 dataset and v9 schema remain maintained for edition 1. Version 9 cannot represent both editions or qualified-answer transcripts. Edition 1.1 results are available in its schema 10 download above.

02 Explore the file

Find the main records.

These four paths cover rankings, runs, trials, and episode transcripts.

leaderboard[]
Ranks, model identity, question score, success, reliability, cost, and time.
official_runs[]
Full selected runs with configuration, totals, subjects, and component models.
subjects[].trials[]
Trial outcomes, counted questions, penalties, cost, duration, and contract status.
trials[].episode
Typed transcripts, public evidence, usage, timing, and visible format violations.

Example query

Print leaderboard rank, model name, and exact question score.

jq -r '.leaderboard[] | [.rank, .model.display_name, .question_score] | @tsv' deep20bench-v10.json

03 Verify and reuse

Check the release details.

Review what went into this build, then cite it and reuse it.

Validated source runs
35
Selected official runs
11
Benchmark protocol
v9
Latest completion
Active cohort
qualified-core-v1
Subject catalog SHA-256
ff2fb52f3aa2213913e9b979211e55cc73b7bb9ae5185b44bb2a312dd1c4965a

Contents

Public and excluded.

Included

IDs, model settings, scores, outcomes, costs, timestamps, source commit, contract reliability, typed transcripts, published Oracle evidence, and source attribution for reused answers. Contract-violation records may include captured visible Guesser text, including invalid output.

Excluded

Adjudicator prompts, raw responses and internal decisions, hidden reasoning, full provider payloads, credentials, headers, sessions, and private subject state. Owner-only diagnostic files are not published.

The publisher reads completed signed run files. It makes no model calls and does not change source runs.

Software licence

Source-available software

The software is source-available under a dual-license model: PolyForm Noncommercial for noncommercial use, with separate commercial licenses available.

Software licensing - source-available ↗