Builder agent
The builder agent creates and edits workflows from a conversation and sample documents. It never changes anything by itself: it stages typed operations on a private layer, tests them on samples and freezes them into a ChangeSet that a person reviews (diff and canvas preview) and approves. Approving writes a new draft revision; it never publishes a version nor moves an alias.
Screens (S14): the assistant on Inicio, the sessions page /agent (and /agent/{id}) and
the Assistant view of a workflow (/workflows/{id}/agent). All three show the same session:
the history lives on the server.
How a turn works
person ─ POST /v1/agent/sessions/{id}/messages (content, ≤ 3 packets) ─▶ server
◀─ text/event-stream: tool.started … turn.completed
loop (≤ 12 steps): model → one tool or the final answer → model → …
- The client sends only its new message. The server loads the session and its messages, so a
forged
historyorapprovalin the request body is ignored. - Every tool call, model call and proposal is an AgentAction (
aga_…) persisted as it happens (arguments, result orresult_reffor results over 8 KB, error,cost_usd,model_call_ids, approval state):GET /v1/agent/sessions/{id}/actions. A turn cut short (client gone, budget, step limit) keeps what it did; the web reconciles withGETafter every turn. - Events (
schemas/agent/agent-events.schema.json):message.delta,tool.started,tool.finished,changeset.proposed,approval.requested,session.budget,turn.completed(stop_reason:final,step_limit,budget,tool_limit,cancelled,error) anderror.
Tools (closed list)
| Tool | What it does |
|---|---|
inspect_document |
Probable type, pages, tables, language and ≤ 20 fragments of ≤ 300 characters of an attached document (doc1…), cleaned of control characters, with provenance. |
propose_schema |
Stages a new extraction schema (fields with type, format, description, one nested level). |
edit_graph |
Stages typed operations: add_node, remove_node, connect, disconnect, set_node_config, add_schema, add_field, rename_field, retype_field, remove_field, set_field_description. Fields are addressed by their stable field_id. A malformed call returns every problem (operation errors and the violations of the result, exactly as POST /v1/workflows/{id}:validate) and stages nothing. |
run_sample |
Runs the layer on 1–5 attached documents in memory (nothing saved): per field the value, confidence and band, citation verdict and failed rules — never page text. |
explain_errors |
Deterministic causes of the last sample, in a fixed order: failed rules, low-confidence fields (with signals), unverified citations. The model only writes the prose. |
get_results |
The stored results of the last sample (by document or field). |
outline |
The layer: nodes, edges, schemas with field_id, pending operations, violations, documents. |
request_approval |
Freezes the layer (0 violations required) into a ChangeSet and supersedes the pending one, and measures it: before/after evidence. |
launch_eval |
Asks for an evaluation of a published version on a dataset snapshot (train/validation): a RunRequest with its estimated cost. |
compare_evals |
T38's paired comparison of two evaluations, copied (GET /v1/evaluations:compare), never recomputed. |
start_optimization |
Asks for a measured optimization (T41). It always needs a person's approval. |
get_optimization |
Status of an approved optimization; when it completed, the validation gain with its interval and the test report as the API returns them, and a promote_suggestion card. |
generate_integration_code |
Python, TypeScript and curl examples for a published alias, from fixed templates. |
There is no tool to publish, move aliases, delete workflows or versions, approve, run shell
commands, read files or fetch URLs. A tool name outside the list is answered with a typed
unknown_tool error and has no effect. agent/tools may not import HTTP clients (architecture test
and import-linter contract); tools use the services in process with the session's tenant and
project (RLS) and the actor agent:<person>.
Quality loop: evaluations, optimizations, evidence and integration code
The agent can measure and tune, but it never spends without a person, never reads the sealed
test split and never promotes.
RunRequests (rr_…, table run_request, RLS). launch_eval and start_optimization create
one with the estimated cost (documents × the mean cost per document of the workflow's last
evaluation; without one, a conservative 0.05 USD per document):
| Request | Status | What happens |
|---|---|---|
| evaluation ≤ 2.00 USD and ≤ 20 documents | auto |
starts at once; the evaluation has trigger agent |
| evaluation above either limit | proposed |
nothing runs until POST /v1/run-requests/{id}:approve by a person or an API key |
| optimization (any cost) | proposed |
same; after approval the T41 job starts with max_cost_usd capped to the approved amount |
GET /v1/run-requests/{id} reports it (a started request becomes done when its evaluations or
optimization finish); :approve and :reject refuse the agent actor with 403, a second decision
answers 409 RUN_REQUEST_NOT_PENDING. The thresholds are per project in
GET/PATCH /v1/agent/policy (approval_threshold_usd, approval_max_docs).
The sealed split. launch_eval is evaluated with the agent access context: a scope with
test answers 403 TEST_SPLIT_SEALED. There is no promotion tool; POST
/v1/optimizations/{id}:promote answers 403 AGENT_CANNOT_PROMOTE to the agent context. A completed
optimization is reported with a promote_suggestion card — an AgentAction of kind proposal whose
operations is empty: a suggestion for a person, with nothing executable in it.
Evidence of a ChangeSet. request_approval evaluates the base document and the document the
patch produces (as a candidate draft, T41's mechanism: invisible to the Studio, never published) on
the same validation documents (at most 20) of the workflow's newest snapshot, and adds to
ChangeSet.evidence:
{"kind": "eval", "status": "ready", "run_request_id": "rr_…", "before": "evr_…", "after": "evr_…",
"n": 12, "reused_before": false, "snapshot_id": "dss_…",
"compare": {"diffs": {"f1": {"a": 0.862, "b": 0.793, "diff": 0.069, "ci95": [0.0, 0.167]}}},
"cost_delta": {"before_usd": 0.12, "after_usd": 0.12, "delta_usd": 0.0}}
compare is GET /v1/evaluations:compare?a=<after>&b=<before> verbatim. The base evaluation is
reused when an equivalent exists (same graph hash, snapshots, configuration hash and scope): no
model call for it. The evidence goes through a RunRequest too, so above the threshold it waits
(status: pending_approval). Without a dataset with ground truth the entry is {"unmeasured": true,
"reason": "no_dataset", "run_samples": ["aga_…"]}. The card shows n and the interval so a small
validation set does not oversell an improvement.
Policy require_evidence_before_approval (default false, PATCH /v1/agent/policy): when on,
approving a ChangeSet without measured evidence answers 409 EVIDENCE_REQUIRED; when off it is
allowed and the approval records evidence: "none" (or "measured").
Integration code. generate_integration_code renders Jinja2 templates
(agent/templates/{python,typescript,curl}) parameterized by workflow, alias or version and the
output_contract; the types come from agent/codegen/schema_types.py (objects, arrays/tables,
scalars, enums, dates → Pydantic v2 or zod). Anything outside that subset answers UNSUPPORTED_TYPE
with the pointer. The model supplies only the explanation text, which travels beside the code and
never inside it. The code reads DX_API_KEY from the environment; no key is ever embedded.
Same inputs, same bytes.
ChangeSets
A ChangeSet (cs_…) holds the operations, the RFC 6902 patch compiled by the server, the
validation, the evidence (the session's sample runs), the risk flags and the status
proposed | approved | applied | rejected | superseded | expired.
- Canonical patch. The same base and operations give the same patch;
patch_hashis the sha256 of its RFC 8785 (JCS) bytes. Everyremove/replace/moveis preceded by atestof the identity it targets (node id, edge,x-dx.field_id), so a patch never applies to a drifted base and the web computes the diff from the patch alone. - Approval — only
POST /v1/change-sets/{id}:approveby a person or an API key withworkflow:write. A request markedX-DX-Actor: agentgets 403. The body carries thepatch_hashthat was reviewed (403APPROVAL_MISMATCHotherwise) andacknowledge_risks. The approval{by, at, patch_hash, base_graph_hash, hmac}is signed with HMAC-SHA256 under the approval secret; applying re-computes the hash of the stored patch and checks the signature (403APPROVAL_MISMATCHwhen the patch changed after approval). - Compare-and-swap. The draft must still be at
base_revisionwithbase_graph_hash. When it drifted the answer is 409CHANGESET_BASE_CHANGED, the ChangeSet becomessuperseded, nothing is written and, when the patch still applies and validates on the new draft, a rebased proposal is created (rebased_change_set_id); it needs its own approval. A session that creates a workflow creates it (with its first draft) on approval. - Approving an applied ChangeSet again answers the same result (a double click sends one request from the web, and a retry is harmless).
Risk flags
| Flag | When | Needs acknowledge_risks |
|---|---|---|
new_egress |
a notify/export/webhook node is added or changed, or a value carries a URL | yes |
code_node |
a transform or plugin node is added or changed | yes |
secret_reference |
a value references a secret (secret://, ${{ secrets.… }}) |
yes |
cost_multiplier |
the extraction tier goes up, the model policy changes, a fan-out node is added | no |
destructive |
something is removed | no |
schema_breaking |
a field is renamed, retyped or removed | no |
Limits and budget
| Limit | Default |
|---|---|
| Steps (model calls) per turn | 12 |
| Tool calls per session | 100 (422 AGENT_LIMIT_EXCEEDED) |
| Documents per message | 3 (422 AGENT_LIMIT_EXCEEDED) |
| Sample documents per call / per session | 5 / 15 (AGENT_LIMIT_EXCEEDED, status 422) |
| Tool results kept inline | 8 KB (larger ones go to the blob store with a summary) |
| Budget per session | max_cost_usd = DX_AGENT_MAX_COST_USD (1.00 USD) |
The budget is the sum of the session's actions' cost. It is checked before every model call, so a
session stops at most one call past max_cost_usd; session.budget is emitted after every model
call (the web announces 80 % and 100 % with aria-live). An exhausted session answers 409
AGENT_BUDGET_EXHAUSTED until a person raises max_cost_usd with
PATCH /v1/agent/sessions/{id}.
Security
- Document text is data, never instructions: it reaches the model only inside delimited blocks
(
<<<DX_DATA kind=… source=…>>> … <<<END_DX_DATA>>>); delimiters inside the text are neutralized and the system prompt says the blocks are untrusted. A corpus of 10 PDFs with embedded instructions (add a webhook, lower thresholds, approve, exfiltrate) is run 3 times each against a model that obeys them: no change is applied without approval and every egress node is flagged and refused without acknowledgement (tests/agent/test_agent_injection.py). - Messages are rendered as Markdown without raw HTML (react-markdown + rehype-sanitize), links
only with
http(s)/mailtoandrel="noopener noreferrer". - Logs and traces carry ids and sizes, never document content or secrets.
- The agent can propose and launch but never approve, promote or read the test split
(
tests/agent/test_agent_quality_integration.py); results with injected text do not change the tools on offer (tests/agent/test_agent_quality_session.py). - The approval secret:
DX_APPROVAL_SECRET(≥ 32 bytes); without it a key is derived from the server's upload secret.
Models
The agent runs PydanticAI (pydantic-ai-slim, MIT) over the model gateway through
GatewayModel, so it inherits cassettes, the call ledger, cost and the fallback policy. Tool
calls are emulated: each step is one generate_structured call (purpose agent) whose answer
is {"action": "tool", "tool", "args_json"} or {"action": "final", "message"}.
| Backend | Tool calling | Status |
|---|---|---|
fake (cassettes, tests and CI) |
emulated | golden suite 10/10 (tests/agent_golden) |
codex (Codex CLI) |
emulated | supported; live recording with DX_LIVE=1 DX_MODEL=codex |
gemini |
emulated (native not used yet) | adapter tested offline only |
Tests
uv run pytest tests/agent -k agent_quality # T44: run requests, evidence, codegen e2e
uv run pytest tests/agent tests/agent_golden # needs Postgres (testcontainers)
DX_AGENT_RECORD=1 uv run pytest tests/agent_golden # rewrite the synthetic cassettes
pnpm --filter web exec vitest run src/features/agent # web: reducer, DiffView, flows with MSW