Measured 2026-08-19
llama3.2:latest
Running locally on Apple M4 · 24 GB unified memory, llama3.2:latest passed 21 of 22 capability tasks (95%), with a median latency of 1,521 ms and 16,052 tokens of context actually read.
Harness 1.1.0 · deterministic graders, no LLM judge · how this is measured
By category
| category | passed | rate | flaky tasks | median latency | what it means |
|---|---|---|---|---|---|
| code | 3/3 | 100% | — | 1,504 ms | Wrote code that executes and returns the expected value. |
| longcontext | 4/4 | 100% | — | 16,326 ms | Actually read a fact buried in a long prompt, rather than truncating it. |
| instruction | 5/5 | 100% | — | 492 ms | Followed a literal instruction exactly (casing, word choice, length). |
| json | 3/4 | 75% | — | 888 ms | Emitted parseable, schema-correct JSON with no prose wrapper. |
| reasoning | 4/4 | 100% | — | 2,608 ms | Got a deterministic answer right — one with a single checkable result. |
| refusal | 2/2 | 100% | — | 8,990 ms | Answered a benign question instead of refusing it spuriously. |
Failed every attempt
- json-extract (json) — price should be 24.5, got 8.17
What this review does not say
Nothing here measures prose quality, creativity, or taste — the graders are deterministic by design, which is what makes the numbers reproducible. A high instruction score does not mean this model writes well. It means it did what it was told, on Apple M4, on 2026-08-19, and you can re-run the suite and check.