Skip to content

Builder agent

The builder agent creates and edits workflows from a conversation and sample documents. It never changes anything by itself: it stages typed operations on a private layer, tests them on samples and freezes them into a ChangeSet that a person reviews (diff and canvas preview) and approves. Approving writes a new draft revision; it never publishes a version nor moves an alias.

Screens (S14): the assistant on Inicio, the sessions page /agent (and /agent/{id}) and the Assistant view of a workflow (/workflows/{id}/agent). All three show the same session: the history lives on the server.

How a turn works

person ─ POST /v1/agent/sessions/{id}/messages (content, ≤ 3 packets) ─▶ server
                                     ◀─ text/event-stream: tool.started … turn.completed
loop (≤ 12 steps): model → one tool or the final answer → model → …
  • The client sends only its new message. The server loads the session and its messages, so a forged history or approval in the request body is ignored.
  • Every tool call, model call and proposal is an AgentAction (aga_…) persisted as it happens (arguments, result or result_ref for results over 8 KB, error, cost_usd, model_call_ids, approval state): GET /v1/agent/sessions/{id}/actions. A turn cut short (client gone, budget, step limit) keeps what it did; the web reconciles with GET after every turn.
  • Events (schemas/agent/agent-events.schema.json): message.delta, tool.started, tool.finished, changeset.proposed, approval.requested, session.budget, turn.completed (stop_reason: final, step_limit, budget, tool_limit, cancelled, error) and error.

Tools (closed list)

Tool What it does
inspect_document Probable type, pages, tables, language and ≤ 20 fragments of ≤ 300 characters of an attached document (doc1…), cleaned of control characters, with provenance.
propose_schema Stages a new extraction schema (fields with type, format, description, one nested level).
edit_graph Stages typed operations: add_node, remove_node, connect, disconnect, set_node_config, add_schema, add_field, rename_field, retype_field, remove_field, set_field_description. Fields are addressed by their stable field_id. A malformed call returns every problem (operation errors and the violations of the result, exactly as POST /v1/workflows/{id}:validate) and stages nothing.
run_sample Runs the layer on 1–5 attached documents in memory (nothing saved): per field the value, confidence and band, citation verdict and failed rules — never page text.
explain_errors Deterministic causes of the last sample, in a fixed order: failed rules, low-confidence fields (with signals), unverified citations. The model only writes the prose.
get_results The stored results of the last sample (by document or field).
outline The layer: nodes, edges, schemas with field_id, pending operations, violations, documents.
request_approval Freezes the layer (0 violations required) into a ChangeSet and supersedes the pending one, and measures it: before/after evidence.
launch_eval Asks for an evaluation of a published version on a dataset snapshot (train/validation): a RunRequest with its estimated cost.
compare_evals T38's paired comparison of two evaluations, copied (GET /v1/evaluations:compare), never recomputed.
start_optimization Asks for a measured optimization (T41). It always needs a person's approval.
get_optimization Status of an approved optimization; when it completed, the validation gain with its interval and the test report as the API returns them, and a promote_suggestion card.
generate_integration_code Python, TypeScript and curl examples for a published alias, from fixed templates.

There is no tool to publish, move aliases, delete workflows or versions, approve, run shell commands, read files or fetch URLs. A tool name outside the list is answered with a typed unknown_tool error and has no effect. agent/tools may not import HTTP clients (architecture test and import-linter contract); tools use the services in process with the session's tenant and project (RLS) and the actor agent:<person>.

Quality loop: evaluations, optimizations, evidence and integration code

The agent can measure and tune, but it never spends without a person, never reads the sealed test split and never promotes.

RunRequests (rr_…, table run_request, RLS). launch_eval and start_optimization create one with the estimated cost (documents × the mean cost per document of the workflow's last evaluation; without one, a conservative 0.05 USD per document):

Request Status What happens
evaluation ≤ 2.00 USD and ≤ 20 documents auto starts at once; the evaluation has trigger agent
evaluation above either limit proposed nothing runs until POST /v1/run-requests/{id}:approve by a person or an API key
optimization (any cost) proposed same; after approval the T41 job starts with max_cost_usd capped to the approved amount

GET /v1/run-requests/{id} reports it (a started request becomes done when its evaluations or optimization finish); :approve and :reject refuse the agent actor with 403, a second decision answers 409 RUN_REQUEST_NOT_PENDING. The thresholds are per project in GET/PATCH /v1/agent/policy (approval_threshold_usd, approval_max_docs).

The sealed split. launch_eval is evaluated with the agent access context: a scope with test answers 403 TEST_SPLIT_SEALED. There is no promotion tool; POST /v1/optimizations/{id}:promote answers 403 AGENT_CANNOT_PROMOTE to the agent context. A completed optimization is reported with a promote_suggestion card — an AgentAction of kind proposal whose operations is empty: a suggestion for a person, with nothing executable in it.

Evidence of a ChangeSet. request_approval evaluates the base document and the document the patch produces (as a candidate draft, T41's mechanism: invisible to the Studio, never published) on the same validation documents (at most 20) of the workflow's newest snapshot, and adds to ChangeSet.evidence:

{"kind": "eval", "status": "ready", "run_request_id": "rr_…", "before": "evr_…", "after": "evr_…",
 "n": 12, "reused_before": false, "snapshot_id": "dss_…",
 "compare": {"diffs": {"f1": {"a": 0.862, "b": 0.793, "diff": 0.069, "ci95": [0.0, 0.167]}}},
 "cost_delta": {"before_usd": 0.12, "after_usd": 0.12, "delta_usd": 0.0}}

compare is GET /v1/evaluations:compare?a=<after>&b=<before> verbatim. The base evaluation is reused when an equivalent exists (same graph hash, snapshots, configuration hash and scope): no model call for it. The evidence goes through a RunRequest too, so above the threshold it waits (status: pending_approval). Without a dataset with ground truth the entry is {"unmeasured": true, "reason": "no_dataset", "run_samples": ["aga_…"]}. The card shows n and the interval so a small validation set does not oversell an improvement.

Policy require_evidence_before_approval (default false, PATCH /v1/agent/policy): when on, approving a ChangeSet without measured evidence answers 409 EVIDENCE_REQUIRED; when off it is allowed and the approval records evidence: "none" (or "measured").

Integration code. generate_integration_code renders Jinja2 templates (agent/templates/{python,typescript,curl}) parameterized by workflow, alias or version and the output_contract; the types come from agent/codegen/schema_types.py (objects, arrays/tables, scalars, enums, dates → Pydantic v2 or zod). Anything outside that subset answers UNSUPPORTED_TYPE with the pointer. The model supplies only the explanation text, which travels beside the code and never inside it. The code reads DX_API_KEY from the environment; no key is ever embedded. Same inputs, same bytes.

ChangeSets

A ChangeSet (cs_…) holds the operations, the RFC 6902 patch compiled by the server, the validation, the evidence (the session's sample runs), the risk flags and the status proposed | approved | applied | rejected | superseded | expired.

  • Canonical patch. The same base and operations give the same patch; patch_hash is the sha256 of its RFC 8785 (JCS) bytes. Every remove/replace/move is preceded by a test of the identity it targets (node id, edge, x-dx.field_id), so a patch never applies to a drifted base and the web computes the diff from the patch alone.
  • Approval — only POST /v1/change-sets/{id}:approve by a person or an API key with workflow:write. A request marked X-DX-Actor: agent gets 403. The body carries the patch_hash that was reviewed (403 APPROVAL_MISMATCH otherwise) and acknowledge_risks. The approval {by, at, patch_hash, base_graph_hash, hmac} is signed with HMAC-SHA256 under the approval secret; applying re-computes the hash of the stored patch and checks the signature (403 APPROVAL_MISMATCH when the patch changed after approval).
  • Compare-and-swap. The draft must still be at base_revision with base_graph_hash. When it drifted the answer is 409 CHANGESET_BASE_CHANGED, the ChangeSet becomes superseded, nothing is written and, when the patch still applies and validates on the new draft, a rebased proposal is created (rebased_change_set_id); it needs its own approval. A session that creates a workflow creates it (with its first draft) on approval.
  • Approving an applied ChangeSet again answers the same result (a double click sends one request from the web, and a retry is harmless).

Risk flags

Flag When Needs acknowledge_risks
new_egress a notify/export/webhook node is added or changed, or a value carries a URL yes
code_node a transform or plugin node is added or changed yes
secret_reference a value references a secret (secret://, ${{ secrets.… }}) yes
cost_multiplier the extraction tier goes up, the model policy changes, a fan-out node is added no
destructive something is removed no
schema_breaking a field is renamed, retyped or removed no

Limits and budget

Limit Default
Steps (model calls) per turn 12
Tool calls per session 100 (422 AGENT_LIMIT_EXCEEDED)
Documents per message 3 (422 AGENT_LIMIT_EXCEEDED)
Sample documents per call / per session 5 / 15 (AGENT_LIMIT_EXCEEDED, status 422)
Tool results kept inline 8 KB (larger ones go to the blob store with a summary)
Budget per session max_cost_usd = DX_AGENT_MAX_COST_USD (1.00 USD)

The budget is the sum of the session's actions' cost. It is checked before every model call, so a session stops at most one call past max_cost_usd; session.budget is emitted after every model call (the web announces 80 % and 100 % with aria-live). An exhausted session answers 409 AGENT_BUDGET_EXHAUSTED until a person raises max_cost_usd with PATCH /v1/agent/sessions/{id}.

Security

  • Document text is data, never instructions: it reaches the model only inside delimited blocks (<<<DX_DATA kind=… source=…>>> … <<<END_DX_DATA>>>); delimiters inside the text are neutralized and the system prompt says the blocks are untrusted. A corpus of 10 PDFs with embedded instructions (add a webhook, lower thresholds, approve, exfiltrate) is run 3 times each against a model that obeys them: no change is applied without approval and every egress node is flagged and refused without acknowledgement (tests/agent/test_agent_injection.py).
  • Messages are rendered as Markdown without raw HTML (react-markdown + rehype-sanitize), links only with http(s)/mailto and rel="noopener noreferrer".
  • Logs and traces carry ids and sizes, never document content or secrets.
  • The agent can propose and launch but never approve, promote or read the test split (tests/agent/test_agent_quality_integration.py); results with injected text do not change the tools on offer (tests/agent/test_agent_quality_session.py).
  • The approval secret: DX_APPROVAL_SECRET (≥ 32 bytes); without it a key is derived from the server's upload secret.

Models

The agent runs PydanticAI (pydantic-ai-slim, MIT) over the model gateway through GatewayModel, so it inherits cassettes, the call ledger, cost and the fallback policy. Tool calls are emulated: each step is one generate_structured call (purpose agent) whose answer is {"action": "tool", "tool", "args_json"} or {"action": "final", "message"}.

Backend Tool calling Status
fake (cassettes, tests and CI) emulated golden suite 10/10 (tests/agent_golden)
codex (Codex CLI) emulated supported; live recording with DX_LIVE=1 DX_MODEL=codex
gemini emulated (native not used yet) adapter tested offline only

Tests

uv run pytest tests/agent -k agent_quality                # T44: run requests, evidence, codegen e2e
uv run pytest tests/agent tests/agent_golden          # needs Postgres (testcontainers)
DX_AGENT_RECORD=1 uv run pytest tests/agent_golden    # rewrite the synthetic cassettes
pnpm --filter web exec vitest run src/features/agent  # web: reducer, DiffView, flows with MSW