Skip to content

Serving a local model: what was measured

Setup: Apple M4 Pro, 64 GB, llama.cpp server, Qwen3.8-27B (MTP, Q4_K_M), one request at a time. Task: summarise a 3 KB text in 5 bullets with line citations (same prompt every time, 2 runs). Numbers are from one machine and few runs: they show direction, not precision.

Configuration Wall time Output tokens Generation speed
Context 262,144, prompt cache 26 GB, reasoning auto 86-93 s 1,183-1,192 13.9 tok/s
Context 32,768, prompt cache 2 GB, reasoning auto 99-181 s 1,386-3,000 16.6 tok/s
Same, MTP draft 3, micro-batch 512 129-174 s 2,032-3,000 15.9 tok/s
Context 32,768, cache 2 GB, reasoning budget 300 44-58 s 654-707 14.8 tok/s
Same, reasoning off 29-37 s 442-457 15.4 tok/s

What it says

  • A smaller context and cache help a little (+19% generation speed; free memory 6.5 GB instead of 0.1 GB), not enough on their own.
  • The time is the number of output tokens. With reasoning on, a 5-bullet summary produced 1,200 to 3,000 tokens (once it hit the 3,000 cap). Capping reasoning is the big lever: 2-4x faster.
  • Draft length 3 and a smaller micro-batch did not help.
  • Prefill is slow (about 60 tokens/s), so large inputs cost minutes; chunking and not re-reading matter more than any flag.

Consequence for deputy

Reasoning is a per-task-class parameter, not a global one: off or small for extraction and translation, larger for code. The router sets it from measured profiles. Quality at each setting must be measured by the benchmark before it becomes a default; speed alone is not a reason.

Nightly benchmark (macOS)

Regressions in a model's behaviour are invisible to ordinary CI, which has no model. To watch yours, run the benchmark every night on the machine that serves it. This uses launchd on your own Mac, not a CI runner: a self-hosted runner on a public repository would run code from pull requests.

mkdir -p ~/.local/share/deputy ~/Library/LaunchAgents
cp examples/nightly-bench.sh ~/.local/share/deputy/
cp examples/deputy-nightly-bench.plist ~/Library/LaunchAgents/dev.deputy.nightly-bench.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/dev.deputy.nightly-bench.plist

It runs at 03:00, skips (and says so in skipped.log) when the backend does not answer, and keeps one report per night in ~/.local/state/deputy/nightly/ for 90 days. Reports never block anything: compare two nights with deputy bench --compare BEFORE.json AFTER.json. Remove it with launchctl bootout gui/$(id -u)/dev.deputy.nightly-bench.

Comparing prints the pass rate, time per task and estimated saving of both reports, names the tasks that failed more (or less) and says if the served model changed. It exits 0 unless you add --fail-on-regression, so a cron job can alert on it. Only a pass rate that fell by more than 5 points counts as a regression (one extra failure among three repetitions is noise); tasks that failed more or less are listed as changes; per-task detail is limited to failure counts because a report keeps no per-task pass rate.