Cost and quality

Efficiency 1

Compare question quality with the recorded cost of the model under test.

Models: 17
Guesser cost range: 59×
Pareto-efficient: 6 of 17
Direction: Lower is better

Official efficiency ranking

Distance from the lower-left ideal.

Question score and Guesser cost are each normalized from 0 to 1 across this cohort. The ranking measures equal-weight distance from their combined minimum. Lower is better.

Normalized distance · lower is betterSelect a model row to view its full run

Trade-off map

Normalized cost and question score.

Both axes use the same 0-to-1 scale. Dashed curves mark equal distance from the lower-left ideal.

Diamond Pareto-efficient - no other model is both cheaper and better Circle Other ranked model

Lower-left is betterCurves show equal ideal distance

gpt-oss-120B (high)0.096 distance · Pareto-efficient
Claude Opus 5 (high)0.127 distance · Pareto-efficient
Gemini 3.7 Flash (high)0.151 distance
Claude Sonnet 5 (high)0.157 distance
Grok 4.6 (high)0.160 distance
Grok 4.5 (high)0.160 distance · Pareto-efficient
GPT-5 Nano (medium)0.170 distance · Pareto-efficient
Gemini 3.8 Flash (high)0.218 distance
GPT-5.6 Luna (high)0.283 distance
Claude Fable 5 (high)0.343 distance · Pareto-efficient
GPT-5.6 Sol (high)0.350 distance
Kimi K3 (high)0.396 distance
Gemini 3.6 Flash (high)0.407 distance
GPT-6 Astra (high)0.665 distance
Mistral Medium 3.5 (high)0.908 distance
Llama 4 Maverick (non-thinking)1.000 distance · Pareto-efficient
Claude Fable 5.1 (high)1.000 distance

gpt-oss-120B (high): 13.63 questions, $0.0363 Guesser cost per episode, ideal distance 0.096, rank 1. Pareto-efficient. View full run for gpt-oss-120B (high)
Claude Opus 5 (high): 12.34 questions, $0.0705 Guesser cost per episode, ideal distance 0.127, rank 2. Pareto-efficient. View full run for Claude Opus 5 (high)
Gemini 3.7 Flash (high): 14.03 questions, $0.0653 Guesser cost per episode, ideal distance 0.151, rank 3. View full run for Gemini 3.7 Flash (high)
Claude Sonnet 5 (high): 14.74 questions, $0.0493 Guesser cost per episode, ideal distance 0.157, rank 4. View full run for Claude Sonnet 5 (high)
Grok 4.6 (high): 14.29 questions, $0.0653 Guesser cost per episode, ideal distance 0.160, rank 5. View full run for Grok 4.6 (high)
Grok 4.5 (high): 15.17 questions, $0.0286 Guesser cost per episode, ideal distance 0.160, rank 6. Pareto-efficient. View full run for Grok 4.5 (high)
GPT-5 Nano (medium): 15.49 questions, $0.0138 Guesser cost per episode, ideal distance 0.170, rank 7. Pareto-efficient. View full run for GPT-5 Nano (medium)
Gemini 3.8 Flash (high): 13.14 questions, $0.1128 Guesser cost per episode, ideal distance 0.218, rank 8. View full run for Gemini 3.8 Flash (high)
GPT-5.6 Luna (high): 17.74 questions, $0.0184 Guesser cost per episode, ideal distance 0.283, rank 9. View full run for GPT-5.6 Luna (high)
Claude Fable 5 (high): 12.06 questions, $0.1776 Guesser cost per episode, ideal distance 0.343, rank 10. Pareto-efficient. View full run for Claude Fable 5 (high)
GPT-5.6 Sol (high): 14.40 questions, $0.1711 Guesser cost per episode, ideal distance 0.350, rank 11. View full run for GPT-5.6 Sol (high)
Kimi K3 (high): 12.74 questions, $0.2032 Guesser cost per episode, ideal distance 0.396, rank 12. View full run for Kimi K3 (high)
Gemini 3.6 Flash (high): 16.89 questions, $0.1705 Guesser cost per episode, ideal distance 0.407, rank 13. View full run for Gemini 3.6 Flash (high)
GPT-6 Astra (high): 15.09 questions, $0.3279 Guesser cost per episode, ideal distance 0.665, rank 14. View full run for GPT-6 Astra (high)
Mistral Medium 3.5 (high): 20.06 questions, $0.4113 Guesser cost per episode, ideal distance 0.908, rank 15. View full run for Mistral Medium 3.5 (high)
Llama 4 Maverick (non-thinking): 32.23 questions, $0.0086 Guesser cost per episode, ideal distance 1.000, rank 16. Pareto-efficient. View full run for Llama 4 Maverick (non-thinking)
Claude Fable 5.1 (high): 12.11 questions, $0.5014 Guesser cost per episode, ideal distance 1.000, rank 17. View full run for Claude Fable 5.1 (high)

Efficiency ranking
Ideal-distance rank	Model	Idealdistance	Paretoefficient	Questionrank	Questionscore	Guesser costper episode	Success
1	gpt-oss-120B (high)cerebras	0.096	Yes	6	13.63	$0.0363	94%
2	Claude Opus 5 (high)anthropic	0.127	Yes	3	12.34	$0.0705	100%
3	Gemini 3.7 Flash (high)google-ai-studio	0.151		7	14.03	$0.0653	97%
4	Claude Sonnet 5 (high)anthropic	0.157		10	14.74	$0.0493	100%
5	Grok 4.6 (high)xai	0.160		8	14.29	$0.0653	100%
6	Grok 4.5 (high)xai	0.160	Yes	12	15.17	$0.0286	94%
7	GPT-5 Nano (medium)openai	0.170	Yes	13	15.49	$0.0138	94%
8	Gemini 3.8 Flash (high)google-ai-studio	0.218		5	13.14	$0.1128	100%
9	GPT-5.6 Luna (high)openai	0.283		16	17.74	$0.0184	91%
10	Claude Fable 5 (high)anthropic	0.343	Yes	1	12.06	$0.1776	100%
11	GPT-5.6 Sol (high)openai	0.350		9	14.40	$0.1711	100%
12	Kimi K3 (high)moonshotai	0.396		4	12.74	$0.2032	94%
13	Gemini 3.6 Flash (high)google-vertex	0.407		14	16.89	$0.1705	97%
14	GPT-6 Astra (high)openai	0.665		11	15.09	$0.3279	94%
15	Mistral Medium 3.5 (high)mistral	0.908		17	20.06	$0.4113	89%
16	Llama 4 Maverick (non-thinking)parasail	1.000	Yes	18	32.23	$0.0086	49%
17	Claude Fable 5.1 (high)anthropic	1.000		2	12.11	$0.5014	97%