03 benchmarks
numbers that reproduce.
every figure on this page is generated from the same public repository the page links to — same procedure, same prompt, same hardware. if the repo changes, this page changes with it. nothing here is hand-typed, and nothing here is a vendor spec sheet.
reading the benchmark data…
models tested
| model | size | quant | measured | status |
|---|
findings
why it lands this way
key insights
known bottlenecks
methodology
hardware & software baseline
the test prompt — identical for every model
metrics we care about
test procedure
fair comparison rules
what we don't test (yet)
known limitations
why this page exists: most "benchmark" content online is a screenshot of someone's best run. This one is generated from a public repo at a pinned commit, so you can read the raw data, re-run the script, and check the procedure yourself. The page shows the commit it was generated from — if that commit is not the tip of the repo, the repo has moved on since this snapshot was taken. The first three figures are that study's conclusions on that date; the live now cell is what the fleet is serving at this moment, and the full picture is on /status.