Methodology
How this is measured
A benchmark is only worth reading if you can see how it was produced. This page describes the harness, the decisions that shape the numbers, and the things these results cannot tell you.
Cost
The invoice, not the price list
Cost is never computed from a published price table. After every call, the harness asks the provider what it actually billed for that specific generation, and records that figure against the run.
This matters because a price-table calculation misses reasoning tokens you were charged for but never saw, cache discounts you did receive, per-request fees, and the difference between the provider you assumed and the one you were routed to. It is wrong in exactly the cases where the number matters.
Where a run could not be reconciled against the provider’s record, it is counted and shown as such on the experiment page rather than quietly averaged in.
Speed
Measured at the stream
Time to first token is taken from the response stream itself, not from a client library that may buffer, retry or normalise before handing anything back. Agent frameworks are excellent at orchestration and quietly ruinous for timing, so the measurement layer sits underneath them.
Every speed figure is a percentile. A mean would hide the tail, and the tail is the part that makes a model unusable in production. Where a run was retried, the retries are excluded from its timings.
Quality
Scored blind, by more than one judge
Outputs are scored against a rubric by judge models that are not told which model produced what. Each judgement breaks into named dimensions before an overall mark, so a model that is faithful but ignores the brief is visibly different from one that is neither.
Judging is run with more than one judge, from different families, because a single judge shows self-preference bias. Where the judges disagree materially — more than about 0.8 on a 1–5 scale — the ordering is noise and the experiment page says so.
Judges currently used: claude-sonnet-4.5, gemini-2.5-pro.
Alongside the judged score, each task runs deterministic checks that need no model at all — word limits, entity recall, whether the answer is in the requested shape. Those carry none of a judge’s bias, and where the two disagree, the deterministic check is the one to trust.
The grid
Same work, repeated
Each experiment is a grid: every case, run by every model, repeated several times. The order is shuffled so a provider having a bad minute does not land entirely on one model. Results append to disk as they complete, so a sweep can resume rather than restart.
The same harness runs single-turn tasks and multi-step agents. For an agent, the unit of measurement is the whole episode: every generation in the trace is billed and counted, not just the final one.
Limits
What these numbers cannot tell you
Note: A benchmark is not your workload
These tasks are fixed and public. Your prompts are longer, your inputs messier and your quality bar different. Treat the ranking as a shortlist, not a decision.
Note: Results are a snapshot
Providers change routing, pricing and capacity. A figure measured on one day is evidence about that day. Every result here is stamped with when it ran.
Note: The grids are small
A handful of cases and repeats is enough to separate models that differ a lot. It is not enough to split hairs between two that sit close together, and percentiles from a small sample are indicative of the tail rather than a precise measure of it.
Note: A judge is a model
Model-graded quality correlates with human judgement well enough to be useful and badly enough to be argued with. That is why every output and every rationale is published — so you can disagree with a specific score rather than the whole exercise.
Reproducing it
The harness is the specification
Every experiment page carries the exact configuration it ran with: the models, the cases, the repeat count, the sampling parameters and the judge setup. The numbers on these pages are generated directly from the recorded runs — there is no hand-curated layer between the measurements and what you see.
If a figure here does not match what you observe, that gap is itself interesting. Tell us.