ADR-012: the builder agent measures, but a person decides
Status: accepted (T43 introduced the agent; T44 adds the quality loop).
Context
The builder agent (T43) proposes ChangeSets that a person approves. T41 optimizes field descriptions on measured data and T38 evaluates versions on managed datasets. The agent should be able to use both โ launch evaluations, start optimizations, attach before/after evidence to its proposals โ without becoming a way to spend money, read the sealed test split or change production.
Decisions
- Approval by cost.
launch_evalcreates a RunRequest with an estimated cost. At or under 2.00 USD and 20 documents (per project:approval_threshold_usd,approval_max_docs) the evaluation starts (auto); above either, it isproposedand nothing runs until a person (or an API key withworkflow:write) approves. An optimization always needs approval, and itsmax_cost_usdnever exceeds the approved amount. Theagentactor gets 403 on every decision. - The agent never reads
test. Its evaluations run with theagentaccess context; a scope withtestis 403TEST_SPLIT_SEALED. The agent has no promotion tool andPOST /v1/optimizations/{id}:promoterefuses its context (AGENT_CANNOT_PROMOTE); a finished optimization is reported with apromote_suggestioncard that carries no operations. The test result it quotes is the one the API returns (the optimizer's single logged read). - Evidence, not opinion. A ChangeSet may carry
{before, after, compare, cost_delta}: two evaluations of the same snapshot on at most 20 validation documents, the comparison being T38's paired one, copied. The base evaluation is reused by (graph hash, snapshots, config hash, scope). The draft is evaluated as a candidate draft (invisible, never published). Without ground truth it isunmeasured. The UI shows n and the interval. - Policy
require_evidence_before_approval(project, defaultfalse): on, approving without measured evidence is 409EVIDENCE_REQUIRED; off, it is allowed and recorded asapproval.evidence = "none". Unmeasured evidence does not satisfy the policy. - Code is never written by the model. Integration examples (Python, TypeScript, curl) come from
deterministic templates and types generated from the
output_contract; the model writes only the explanation. Keys are read fromDX_API_KEY, never embedded.
Consequences
- The agent's spend is bounded by what a person approves; its tools cannot reach the test split.
- A small validation set gives wide intervals: the card says so instead of claiming a gain.
- The ChangeSet contract (06 ยง14) widens
evidence[]from run references to measures; readers that only knewrun_samplereferences keep working (those entries are unchanged).