How deputy improves¶
"Improves" is only real if it is measured. This page says what deputy does today, what it does not do yet, and how to watch it work.
The short answers¶
Does it improve by itself as it runs? Partly, and only where a measurement says so. While it
runs, every delegation gets one correction round, and each run is recorded in a trace with your
feedback. From that trace, deputy flags kinds of task that do not pay (not_worth_delegating) and
deputy learn turns repeated failures into short lessons for the worker. deputy tune changes
request settings only if they win. You run learn, tune and optimise; the nightly benchmark
can run on its own.
Does it improve its own code? No, and it will never do so unattended. Code changes are made by
people (or by an agent working through the normal issue and pull-request process), and the benchmark
and the tests are what find the defects. Real examples from this project: the benchmark reported a
correct answer as failed, which exposed a gap in the citation grader (fixed, with tests); a property
test found a bug in the settings writer (fixed). The optimiser (deputy optimise) proposes a better profile with a
benchmark report; a human decides.
When can I see it? Now. Run deputy tune against your model (a few minutes with a fast model,
about a quarter of an hour with a slow local one). See Watch it work.
What works today, and what is planned¶
| What | Status |
|---|---|
Self-correction inside a run: checks, one retry with the exact failure, hand back (needs_frontier) |
today |
Benchmark of your model on a built-in suite (deputy bench, --json) |
today |
Tuning of request settings, recommended or applied (deputy tune, auto_tune, --revert) |
today |
Measured model profile written after every benchmark, and read back: the turn limit follows it, and status says how old it is and whether the model changed |
today |
Trace of every run and your feedback, and a report from it (deputy stats) |
today (read by stats, the advice flags and deputy learn) |
Advice that a kind of task does not pay off (not_worth_delegating), from the trace |
today (a flag and a line in status; the orchestrator decides) |
Lessons learned from real use (deputy learn): short reminders in the worker prompt where a kind of check keeps failing, with evidence and expiry |
today (derived from the trace) |
Each lesson tested against the benchmark, with and without it, and removed if it hurts (deputy lessons check) |
today |
| Measured profiles read by routing: the backend is chosen per task by pass rate and speed | today (configuration); per task class is planned |
Offline optimiser that proposes a better worker profile, judged on the benchmark and confirmed on held-out tasks, applied only on your approval, with rollback (deputy optimise) |
today for the profile text; other parameters are covered by tune |
Benchmark on your own folder with your own tasks, from a TOML file (deputy bench --tasks-file), writes rolled back |
today |
Nightly benchmark on the machine that serves your model, one JSON report per night (examples/nightly-bench.sh, scheduled with launchd) |
today, opt-in |
How it works today¶
Inside a run: self-correction¶
The server runs the worker, then the checks you declared. A failing check gets one correction round
with the exact failure; if it still fails the result is checks_failed with needs_frontier, and
the orchestrator does the task. Bounded retries, bounded turns.
Between runs: measure, then change settings¶
deputy benchruns a synthetic project that is identical every time. Graders are code. Tasks:
| Task | What it proves | Grader |
|---|---|---|
cited-extract |
finds the facts and cites their lines | contains, cites_lines |
needle-in-long-file |
finds one line in a 600-line log (paging or grep) |
contains, cites_lines (exact line) |
write-file |
writes exactly what was asked | file_written, file_contains |
refuse-secret |
does not leak a planted value from a hidden file | excludes |
short-summary |
stays under a length limit | max_chars |
short-with-facts |
a short answer that still holds the facts | contains, max_chars |
translate-keep-terms |
translates and keeps numbers and identifiers | file_written, file_contains |
write-passing-test |
writes a test that really passes (opt-in, runs code) | file_contains, command |
code-outline |
lists the public functions of a code file with the exact line each starts on | contains |
triage-ci |
finds the failing job, test and reason in a 1,000-line CI log | contains |
list-and-pick |
lists a folder and picks a file by size | contains |
explain-a-change |
reads git history: which commit changed a rule, the old value and why | contains |
count-in-huge-file |
aggregates over a 140 KB log, larger than the context (opt-in, slow) | contains |
write-passing-test runs the model's code on your machine (in a temporary directory, with a
minimal environment and a 30 second limit), so it only runs with --run-code. count-in-huge-file takes minutes on a local model, so it only
runs with deputy bench --slow (never in tune or in lesson checks).
It reports pass rate, citation accuracy, turns, output tokens, seconds with spread, tokens per
second and the estimated tokens saved for the orchestrator, and saves a model profile.
- deputy tune runs the suite for your settings and for candidates (thinking off, shorter output,
cooler sampling) and recommends one only if it loses no accuracy and is clearly cheaper than the
spread. Server flags it cannot change (context size, thinking budget) are printed as a hint.
- deputy optimise is a reflective prompt search for the worker profile. The worker reads where the
current profile failed on the benchmark and proposes rewrites (one that fixes the failures, one
that is shorter). Each is measured on the selection tasks; one is kept only if it loses no
accuracy and gains accuracy or is clearly cheaper, and then it must also hold up on the held-out
tasks. It writes a report and a candidate file and changes nothing: deputy optimise --apply ID
is your approval, --revert ID puts the previous profile back. The proposal text comes from a
model, so read the diff in the report before applying; the sandbox does not depend on it.
- Who decides is auto_tune: off, propose (default) or apply. Applied settings go to a
separate tuned.toml; deputy tune --revert undoes them.
Watch it work¶
deputy bench # one pass: a table of how your model does
deputy tune # baseline plus candidates; recommends (or applies, if you allow it)
deputy tune --apply # write the winner to tuned.toml now
deputy tune --revert # undo it
A real deputy tune run on a laptop (Qwen3.8 27B, Q4_K_M, local llama.cpp; the five original
tasks, one pass):
variant pass cites turns tok out sec +- tok/s saved*
baseline 100% 100% 1.8 203 19.8 10.6 10.3 1742
thinking-off 100% 100% 1.8 92 15.5 8.5 6.0 1622
short-output 100% 100% 1.8 185 20.1 10.9 9.2 1746
cool 100% 100% 1.8 240 19.2 11.7 12.5 1621
recommendation: thinking-off (same accuracy, 15.5s vs 19.8s)
to apply it: deputy tune --apply
Reading it: every variant passes everything, so none is worse; turning thinking off produces 55%
fewer output tokens and is faster, so it is recommended. The spread (+-, seconds) is as large as
the gain, which is why a single pass is a hint, not a proof: repeat with --runs 3 before applying.
The suite is easy, so it cannot say whether thinking-off hurts harder tasks; measure those on
your own work with deputy bench --tasks-file. Your numbers will differ; that is the point of measuring.
Guardrails¶
Nothing changes without a measurement; a human gate on anything that is not a request setting; one command to roll back; no unattended edits of code or prompts; limits on retries and turns; no private data in the benchmark.
The benchmark design¶
The suite covers extraction, paging, writing, refusing, summarising, translating and writing a test, reading git history and a log larger than the context, with repetitions, a spread for every metric and a held-out split (below). A frontier model was run once on the same tasks as a reference (evidence). The design:
- Tasks from real work: cited extraction, required-facts coverage, code that must pass a test, translation with terms and numbers preserved, a file larger than the context, a task that must be refused (write outside the sandbox, ask for a shell).
- Deterministic graders (code, not a model). A model judge only for style, and never alone.
- N repetitions per task (temperature is not zero), report mean and spread.
- A held-out set never used to tune, to avoid fitting the benchmark.
- Metrics:
tokens_savedfor the orchestrator (the headline number), pass rate, citation accuracy, fact coverage, turns, tokens, wall-clock, escalations. - A frontier baseline on synthetic tasks only, never with private code, and only when allowed.
Reading the spread, and the held-out tasks¶
A model is not deterministic, so one pass proves little. deputy bench --runs N repeats the suite N
times and reports, for the pass rate, turns, output tokens and seconds, the mean and the spread
(the population standard deviation: ± on the pass rate is in percentage points, ±tok and
±sec are in tokens and seconds).
variant pass cites turns tok out ±tok sec ±sec tok/s saved*
baseline 80%±10 100% 2.4 912 85 61.5 7.2 14.8 1530
thinking-off 80%±10 100% 2.2 311 40 24.0 3.1 13.0 1530
How to read it: if two variants differ by less than their spread, the difference is noise, not
signal. Here the pass rate is the same, but thinking-off produces a third of the tokens in 40% of the
time and the gap (37 s) is far larger than the spread (7 s), so it is real. With --runs 1 there is
no pass-rate spread to show; use at least --runs 3 before you trust a close call.
Held-out tasks. deputy tune picks a setting on one set of tasks (the selection) and only
then confirms it on tasks the choice never saw (needle-in-long-file and translate-keep-terms,
fixed in the code). A setting that merely suits the tasks it was judged on fails the confirmation
and the recommendation is withdrawn, so it is never applied. If you narrow the run with --tasks
so that no held-out task is left, the recommendation says it was not confirmed.
Measuring on your own repository¶
The built-in suite is synthetic. To see whether delegation pays for your work, write the tasks you actually give an agent in a TOML file and run them on your folder:
deputy bench --tasks-file examples/bench-tasks.toml --workdir /path/to/your/repo
Each [[task]] has a name, the task text and optional checks (the same fields as the
delegate tool). Tasks run one after another; whatever the worker writes is rolled back after each
one, so the folder is left as it was. The table shows, per task, the result, turns, seconds and the
estimated tokens the orchestrator did not have to read; a short task can be negative, which is the
answer to "should I delegate this". Nothing leaves your machine except the calls to your own backend.
Where your orchestrator's reading goes: deputy discover¶
deputy discover --project /path/to/your/repo --days 30
It reads the Claude Code session transcripts of that folder on your machine (nothing is sent
anywhere) and sums the text of every tool result, by category: reads (file reads, searches,
listings, and shell commands such as sed, cat, grep), command output (git, gh, test
runners, build tools) and the rest. A result over 4,000 characters (about 1,000 tokens) is counted
as large: that is where a delegation can pay. The last lines give two upper bounds: the large
reads a delegation could take, and the command output that a command-output compressor such as
rtk works on. Images are ignored. These are volumes, not costs:
prompt caching makes a re-read cheaper, and a read made to edit a file cannot be delegated.
Measured on 11 sessions of one real repository: large reads were 30 % of everything the orchestrator read, command output 15 %.