Model reviews. Show your work.
Public leaderboards benchmark a model as the lab shipped it. That is not the question you have when you are about to run it on your own machine, or route free traffic at a vendor’s copy of it. So we measure the copy in front of us and publish the results file alongside the claim.
Latest run 2026-08-19 · harness 1.1.0 · 22 attempts · 1 model · Apple M4 · 24 GB unified memory
What this can and cannot tell you
Every grader is a pure function of the response text — no LLM judge. Re-run it and you get the same number; read the results file and you can re-derive any verdict yourself. That buys reproducibility, and it costs coverage. Stated plainly:
- Supportable: “best for structured extraction under 8k context” · “passed every instruction task” · “actually read 16k of context”
- Not supportable, and we will not claim it: “best for writing”, “most creative”, “smartest”. Prose quality and taste are not measured here.
- “Could not run” ≠ “ran and got it wrong.” Allocation failures are recorded as their own error class. Collapsing the two would libel the model.
Measured on Apple M4 · 24 GB unified memory
Sample size: 1 model. This is a results table, not a ranking. “Best for X” picks are deliberately withheld until enough models have run for the comparison to mean anything.
| model | overall | measured context | median latency | tok/s | errors |
|---|---|---|---|---|---|
| llama3.2:latest | 21/22 (95%) | 16,052 tok | 1,521 ms | 34.0 | 0% |
Measured context is the largest prompt on which a long-context task actually passed — not the number on the model card. The two are often different, and that gap is the whole point.
What each category means
- instruction — Followed a literal instruction exactly (casing, word choice, length).
- json — Emitted parseable, schema-correct JSON with no prose wrapper.
- longcontext — Actually read a fact buried in a long prompt, rather than truncating it.
- reasoning — Got a deterministic answer right — one with a single checkable result.
- code — Wrote code that executes and returns the expected value.
- refusal — Answered a benign question instead of refusing it spuriously.