Skip to content

Tools

Five tools, short on purpose: the orchestrator pays for their schemas on every turn. A test keeps the total under about 1,500 tokens.

delegate

Run a task on a worker model and verify the result.

Argument Meaning
task What to do. Name files, do not paste them.
workdir Absolute path the worker is confined to.
profile helper by default; see profiles.
checks Proofs the server runs on the answer (below).
backend A configured backend; default is default_backend.
max_turns Safety limit on the worker's tool turns (default 8).
wait true by default. With false it returns at once with status: running; get the result with collect.

Returns: run_id, status (ok, checks_failed, turn_limit, error, and for background runs running, cancelled), answer, a one-line checks summary (3/3 passed, or what failed with its detail), and, only when there are any, files_written and flags (needs_frontier, turn_limit, backend_error, setup_error, time_limit, not_worth_delegating, small_input).

The result is short on purpose. Metrics (turns, tokens, seconds, model) and the saving estimate (orchestrator_tokens_saved_est, a rough bytes / 4 estimate of what you did not have to read) are recorded in the trace and shown by deputy stats, not sent: a typical passing result is about 16 % smaller (558 to 471 tokens in a measured case), and the tool's output schema is smaller too. The worker always answers in English (the bundled profile says so), unless asked to translate.

collect

The result of a wait=False delegation, by its run_id (the same id the result and feedback use). It waits up to 20 seconds, so polling costs few tokens: if the job is still going you get status: running and call it again. cancel=true stops the worker (the request to the model server is dropped and a running command check is killed) and puts back every file the worker wrote, so the workspace is as it was. A result can be collected once. At most 3 jobs run at once.

Jobs live in the memory of the running server: if the server exits, they are gone.

Checks

type Passes when
contains every string in values appears in the answer as a whole, case-insensitive: 12 is not found in 12,599 or 112, log not in logs
excludes none of the strings in values appears in the answer (for example a secret)
file_contains the file path in workdir contains every string in values
cites_text every quoted passage next to a line number (line 2: "the exact words", 2: "...", a.txt:2 "...", or "..." (line 2)) is on that line (or in that range) of the file path; case and spacing do not matter; at least one quoted citation is required
cites_lines every cited line number (line 12, lines 12-14, righe 12, l. 12) is within min..max, and at least one is cited. Give path instead of max and the bound is that file's current number of lines, so the check stays right when the file grows
json_schema the answer (or the file path) is JSON that validates against json_schema (JSON Schema, draft 2020-12); a code fence around the JSON is fine
max_chars the answer is at most limit characters
file_written the worker wrote path
command an allow-listed argv exits 0 in workdir (see configuration)

A failing check gets one correction round with the exact failure. If it still fails, the status is checks_failed with needs_frontier: do the task yourself. The answer of a failed run is cut to its first 300 characters (with a note), because you will redo the work and should not pay to read a wrong answer; the failing checks keep their full detail.

status

Configured backends and whether they answer.

profiles

Available profiles, name and description.

feedback

run_id, verdict (accepted or rejected) and an optional reason. Recorded in the trace; this is what the tuning loop learns from.

not_worth_delegating

deputy keeps count of what each kind of task has done for you (the profile plus the kinds of checks asked for; benchmark runs do not count). Once a kind has been tried at least 5 times, it is flagged when its mean estimated token saving is zero or negative, or when more than half of its runs were handed back to the orchestrator. The flag appears in the delegate result and, with the numbers, in the status tool (advice). The run itself is not refused: the flag tells the orchestrator to do this kind of task itself from now on. The delegation guide says so.

small_input

When every existing file a task names is about a thousand tokens or less (bytes / 4), the result carries small_input with the estimate: reading such a file costs the orchestrator about what the request and the result cost, so next time read it yourself. This needs no history, unlike not_worth_delegating. It never refuses a run, and a task that names no file gets no flag.

Checking what a citation says, not only where it points

cites_lines only proves that the cited numbers are plausible. A worker can cite a real line that says something else. cites_text closes that gap: it finds every quoted passage (at least 8 characters, in double quotes, typographic quotes or backticks, or single quotes) that sits right next to a line reference and checks that the passage is on that line. A quote with no reference next to it is not a citation; one wrong quote fails the check and the detail names the line and the passage. Use both:

{"checks": [
  {"type": "cites_lines", "min": 1, "max": 400},
  {"type": "cites_text", "path": "docs/policy.md"}
]}

and ask the worker, in the task, to quote the exact words of every line it cites. The check is about one file; run one per file.

Listing files: list_files

list_files(path, pattern) returns the files under a folder with their sizes, leaving out what git ignores, node_modules and secret-looking files. pattern is a glob matched against the file name or its path from path (*.md, adr/ADR-*.md). Up to 200 files are listed one per line; more are grouped by folder with a count, so the worker narrows the path instead of reading a long list.

History: read-only git

The worker can read the history of a git repository: git with args such as ["log", "--oneline", "-n", "20"], ["diff", "--stat", "HEAD~3"], ["show", "abc123", "--", "file"] or ["blame", "-L", "10,20", "--", "file"]. Only log, show, diff and blame, with an allow-list of options; paths go after --. The output is capped at 60,000 characters and says when it was cut, and it counts as read on your behalf. Questions such as "what changed in this range and why" or "which commit changed this rule" are long to read and short to answer: good delegations. What is refused, and why, is in security.

Recipe: why did CI fail?

A failed CI run is one of the best things to delegate: the log is long and disposable, the answer is short and checkable. The worker has no network and no credentials, so the orchestrator fetches the log into a file (it pays only for the command, not for the output) and delegates the question:

gh run view <run-id> --log-failed > ci.log     # or --log for the whole run
{"task": "ci.log is the log of a failed CI run. Which job failed, which test or step, and why?",
 "workdir": "/abs/path/to/repo",
 "checks": [{"type": "max_chars", "limit": 1200}]}

Add a contains check for anything you can predict (the job name, say). On the benchmark's 1,000-line CI log (about 18,000 tokens) the local 27B worker named the job, the test and the assertion in 3 runs of 3, in about 2 minutes, with summarise_log and grep.

Large logs: summarise_log

For a big repetitive file (a log, a test run) the worker has a tool that needs no model at all: summarise_log(path) reduces every line to a template (digit runs become #1, #2, ...) and reports each distinct kind of line once: how many lines it stands for, their first and last line number, a real example, and the values of the numbers that vary little (a worker id, a status). Lines that occur at most three times are shown whole with their own line number, so a unique error is never folded away. Only repetition is removed; no kind of line is hidden. A 2,400-line log of 34,500 tokens becomes about 120 tokens, in milliseconds. The worker is told to try it before scan_file, and to read the exact lines it points to.

This is the opposite of a summariser that keeps only lines with a log level: measured on a real CI run of 215 lines, such a filter kept 54 tokens of 7,700 and called the log nearly empty, while summarise_log kept all 176 distinct kinds of line in 3,000 tokens.

Large files: scan_file

A model has a limited context, so a very large file cannot be read in one pass. The worker has a tool for it, scan_file(path, question): the server cuts the file into parts of whole lines (about 24,000 characters each), asks the worker model the question about each part in a fresh conversation, and returns the findings of all parts, each under its range of lines. Every line keeps its number in the original file, so what the worker cites at the end is true of the whole file, and cites_lines and cites_text check it as usual.

Costs, plainly: a scan runs one model call per part (at most 12, so files up to about 290 KB), one after the other, and the whole file is processed. On a local model that is slow at reading its prompt (a 27B model on a laptop reads about 50 tokens a second) 70 KB took about 8 minutes, so a scan has a 10 minute budget and says where it stopped. The worker is told to try grep first, which is fast, and to use scan_file only when grep cannot answer. The orchestrator pays nothing for the parts: the whole file counts as read on its behalf in orchestrator_tokens_saved_est. A grep with up to 50 hits returns them as file:line: text. With more, it returns every matching file once, with its number of hits and its first line numbers, so no file is hidden past a cap (one real search had 66 hits in 29 files: the list costs about 490 tokens, and the worker then greps the file it needs). In a git work tree, grep skips what git ignores (generated folders, mirrored worktrees) and still finds untracked files; elsewhere it searches everything. This only narrows a search: the sandbox's own refusals still apply to each file. A grep, by contrast, counts only the hit lines it returns, because that is all an orchestrator searching by itself would read; counting each matching file would inflate the figure (one search of a large repository once reported 34 million tokens).

Structured answers: json_schema

For a list, a table or any structured answer, ask for JSON and check it against a schema instead of reading it:

{"type": "json_schema", "json_schema": {
  "type": "object", "required": ["rules"],
  "properties": {"rules": {"type": "array", "minItems": 2,
    "items": {"type": "object", "required": ["line", "text"]}}}}}

The answer may be wrapped in a code fence. Add path to validate a file the worker wrote instead. On a mismatch the worker is told exactly what is wrong, with the place inside the document ($.rules[1].line: 'x' is not of type 'integer', $.rules[0]: 'text' is a required property; the first five errors, and how many more). A schema that is not valid JSON Schema is refused up front (invalid check). A $ref to a URL is never fetched: it fails with "unavailable", so the check can never make the server reach out to the network.