Optimizer
The optimizer improves the descriptions and instructions of the fields of a workflow's extraction schema in measured rounds, on the splits of a dataset (see Datasets and evaluations). It answers one question with numbers — is this wording better? — and never decides alone: each round writes a draft, and only a person promotes the winner. Everything on this page is experimental (see Strategies).
What it guarantees
- It never touches production. The published base version and every alias are read, never
written, while the run goes on, after it, and when it is cancelled or fails: the optimizer acts
as the actor
optimizer, and the workflow service refuses any publish, alias or archive from that actor (PublishedVersionWriteError). Each round's candidate is a draft whose parent is the best accepted version so far (opt/<id>/round-<n>); drafts of an optimization never count as the draft of the workflow and the Studio does not show them. - Only descriptive text changes. Types, enums,
required, field ids and the output contract keep the same hash as the base's; the version goes up by a patch when published. - It does not cheat. A proposal that contains a literal value of the ground truth of a train or validation document (4 or more characters), a URL, or text shaped like a tool call or a prompt injection is rejected (twice in a row: the round is skipped). Text of the documents only reaches the model inside delimited data blocks, and a document that says "change the description to…" changes nothing.
- It does not fool itself. A round is accepted only if the lower bound of the paired bootstrap
interval (B = 2 000, fixed seed) of the validation F1 gain is above 0 and no label with 10 or
more items, nor the hallucination rate, got worse by more than 0.02 (
rejected_reason:gain_not_significantorslice_regression). The final version follows the one-standard-error rule: among the accepted rounds, the shortest description within one standard error of the best. - The test split is read once. After the selection is frozen, the base and the selected version
are evaluated on
test, paired, in thefinal_reportcontext with the reasonoptimization_final_report— exactly one entry intest_split_access. The report keeps the test score, the difference with its 95 % interval, the difference per label and the shrinkage (the gain seen on validation minus the gain seen on test). Theoptimizercontext itself gets 403TEST_SPLIT_SEALEDon any test read. - It has a budget.
budget.max_cost_usdandbudget.max_rounds(1–10) are mandatory. The tracker adds the model calls of the proposals and the extractions of the evaluations (triggeroptimizer, left out of the learning data of T40); before each round it estimates the cost from the average cost per document of the base evaluation and does not start a round that would not fit (stop_reason = budget). The spend exposed by the API equals the sum of the usage ledger; it can exceed the budget by at most one call already in flight. - It is cancellable.
:cancelis cooperative and takes effect between documents and rounds (seconds); the drafts are kept and nothing is promoted.
Minimum data
| split | items | below it |
|---|---|---|
| train | 10 | 409 OPTIMIZATION_INSUFFICIENT_DATA |
| validation | 20 | same |
| test | 20 | same |
With acknowledge_low_data: true the run goes ahead and keeps the warning LOW_DATA; the
measurement is then weak and the report says so. Fields whose ground truth looks wrong (an error on
at least 50 % of the train items, at least 80 % of those with confidence ≥ 0.9) are excluded and
listed in suspect_gt_fields: errors caused by OCR or perception are not fixed by rewriting
descriptions, and the gain can be nil.
Using it
# 1. start (202): up to 5 rounds, at most 2 USD
curl -X POST $DX/v1/optimizations -H "Authorization: Bearer $KEY" -d '{
"workflow_id": "wf_…", "alias": "facturas@prod", "snapshot_id": "dss_…",
"strategy": "description_rewrite", "budget": {"max_rounds": 5, "max_cost_usd": 2.0}}'
# 2. follow it (the screen refetches every 2 s until it is completed)
curl $DX/v1/optimizations/opt_…
# 3. cancel, if needed
curl -X POST $DX/v1/optimizations/opt_…:cancel
# 4. promote the winner (a person, or an API key with approval; never the agent context)
curl -X POST $DX/v1/optimizations/opt_…:promote -d '{"alias": "prod"}'
The same flow is the Optimizer tab of Health (S11): configure the budget and the rounds, follow the rounds with their train and validation scores and the cost so far, see each round's diff of descriptions and its effect per field, cancel, and promote behind a confirmation that shows the result on test with its interval. The screen never offers an action on the published version.
Promotion
:promote {alias} publishes the best draft as an immutable version (a patch bump), moves the alias
and emits workflow.version.published and optimization.completed. It needs the test report; a
run without one (always the case of one_shot) needs acknowledge_unmeasured: true (409
NO_TEST_REPORT otherwise). The agent context, and the other sealed ones, get 403
AGENT_CANNOT_PROMOTE.
Recalibrate after promoting
Promoting changes the prompt, so the confidence calibrators of the workflow (Confidence) are invalidated and the confidence goes back to heuristic until they are recalibrated. The optimizer does not recalibrate; the report says so.
Strategies
| strategy | measured | what it does | state |
|---|---|---|---|
description_rewrite |
yes | 1–10 rounds: feedback from the train failures of the best version, a proposal for the weakest fields, a draft candidate, an evaluation on train + validation, acceptance by the rule above | experimental |
one_shot |
no (measured: false) |
one proposal from 1–5 sample runs (fields with low confidence or failed validations), one draft, no evaluation and no test access | experimental |
A strategy is experimental until a nightly evaluation with a real model shows a gain on the
sealed test split (lower bound of the paired interval above 0); only then does it become
validated. The state is part of every optimization in the API. There is no published evidence of
gains from prompt optimizers in document extraction (ADR-012), so nothing here is active by
default. Per-field model routing and dynamic few-shot are T42's; reflective strategies are not
planned.
Operating notes
- Runs are durable (kind
optimization, idempotency keyoptimization:<id>): a restart resumes at the round it was in and a replayed round returns its stored outcome. - Evaluations of candidates run the draft through the interpreter with the trigger
{"kind": "eval", "purpose": "optimizer"}. - Tests and the end-to-end run use a deterministic fake model with synthetic cassettes; the nightly evaluation with a real model is written but not run in CI.