Skip to content

Workflow interpreter

The interpreter (T26) runs a published workflow version durably on the runtime of T14: parse, extract, validate, branch and review nodes, fan-out by page and by scope, a per-node cache, routing by verdict, resume from the failed node, re-run with another version and human review inside the run. Design notes: architecture/interpreter.md.

Running a workflow

POST /v1/runs
{"workflow": "facturas@prod", "packet_id": "pkt_…", "priority": 5, "metadata": {…}}

workflow is name@alias, name@1.2.0, a wfv_… id or {"version_id": "wfv_…"}. Multipart (workflow, files[], metadata, priority) uploads the documents as a new packet first. The alias is resolved once, when the run is created: the run keeps that version even if the alias moves while it runs, waits for a review or is resumed. Errors: unknown alias → 404 ALIAS_NOT_FOUND; a draft or archived version → 422 VERSION_NOT_PUBLISHED; a bare name → 422 ALIAS_REQUIRED. The answer is 202 with the run and Location: /v1/runs/{id}.

Route What it does
GET /v1/runs/{id} (T17) Run status, steps, usage; ?wait=N
GET /v1/runs/{id}/result (T17) Result: outputs (extractions[node_id], validations, classifications, segments, artifacts) and reproducibility (graph_hash, plan_hash, schema_hashes, model_snapshots, engine_versions)
GET /v1/runs/{id}/nodes One entry per (node, scope): status, attempts, cache key and hit, timings, usage, output references
POST /v1/runs/{id}:cancel (T17) Cancels: the node in flight stops at its next check, pending nodes are skipped
POST /v1/runs/{id}:resume (T17) A failed/cancelled run again, same id: only the failed node and its descendants run
POST /v1/runs/{id}:rerun New run (trigger.kind=rerun, parent_run_id) with another version; unchanged nodes come from the cache
POST /v1/runs/{id}/nodes/{node_id}:complete Delivers a human review to a waiting review node; a second call is a no-op
GET /v1/runs/{id}/artifacts/{artifact_id} A node's file, ETag = sha256, 304 on If-None-Match

Node statuses: pending, running, waiting (review), succeeded, failed, skipped (with skip_reason: branch_not_taken, upstream_failed, upstream_skipped, cancelled) and cancelled. Run statuses: succeeded when every active node finished; partially_succeeded when only some scopes (segments, categories, files of a packet) or parallel branches failed; failed when a node without alternative failed (parse, or the extract of a linear workflow); waiting_review while a review waits; cancelled.

The five nodes of T26

Node Runs as What it does
parse child workflow, queue dx_wf_cpu Prepares the file (T17, conversion with T16), fans the pages out in tasks of DX_PARSE_PAGES_PER_TASK pages, at most DX_PARSE_PAGE_CONCURRENCY in flight, and merges the raw pages into one DocIR identical to a single-pass parse. Engines without page ranges (Docling today) degrade to one task per file (warning page_ranges_unsupported).
extract child workflow, queue dx_wf_llm T06 harness + T08 finalize (T17's steps) on the DocIR of the scope (docir_slice), with the version's schema.
validate in the run (inline) or dx_wf_llm/dx_wf_io with AI/external rules T09 rules on the extraction; stores the verdicts, says whether a blocking rule failed.
branch in the run T09 Check on the extraction: pass activates true; fail, inconclusive and a condition that cannot be evaluated activate false.
review in the run policy.when (always, on_blocking_failure, on_low_confidence, on_unclassified); when it applies the node waits (waiting, run waiting_review, event run.waiting_review) until :complete, or fails with REVIEW_TIMEOUT after policy.timeout_hours (168 h by default). The completion (decision, corrections by field path or a whole extraction) becomes the effective extraction downstream; reject fails the node (REVIEW_REJECTED).

parse, extract and validate are cacheable. Classify and split (T28), notify (T27), report and fill_form (T29), transform and plugins (T30), export (T31) add their handlers with the contract below; a run whose plan has a node type without handler fails before running anything.

Handler contract

A handler implements NodeHandler and is registered for its node type:

from typing import Any

from extractit.workflows.contracts import NodeOutcome
from extractit.workflows.interpreter.executor import NodeContext, register_handler


class UppercaseInvoiceNumber:
    """Example: an extraction with the invoice number in upper case."""

    node_type = "transform"
    cacheable = True  # same config + same inputs -> reuse a succeeded node_run
    orchestrates = False  # the whole run() is one durable step with retries

    def run(self, ctx: NodeContext, node: Any) -> NodeOutcome:
        output = ctx.load_extraction()
        number = output.get("data", {}).get("invoice_number")
        if number is None:
            return NodeOutcome.fail("NO_INVOICE_NUMBER", "the extraction has no invoice number")
        output["data"]["invoice_number"] = str(number).upper()
        return NodeOutcome.ok(output_refs={"extraction": ctx.put_json(output)})


register_handler("transform", UppercaseInvoiceNumber(), replace=True)
  • run(ctx, node) -> NodeOutcome gets the typed node of the version (node.config) and a NodeContext: run (ids, version, graph_hash), node_id, scope (key, kind, pages, file_id), inputs (port kind → reference of the nearest upstream instance: parsed, extraction, verdicts…), loaders load_parsed() (the DocIR cut to the scope), load_extraction(), load_verdicts(), put_json()/read_json() (immutable blobs), artifacts.put(bytes | path, kind, media_type, name), usage, logger and cancelled() / check_cancelled() (check it between pages and model calls).
  • NodeOutcome: status (succeeded/failed), output_refs (references, never document content), activated_branches (branching nodes), scopes (fan-out: one ScopeSpec{key, kind file|segment|category, branch?, pages?} per descendant execution), artifacts, warnings, error{code, message}, internal_error. An exception becomes a failed outcome (<CLASS>_ERROR) after the retries; a cancellation propagates.
  • orchestrates = True runs run() in the workflow itself, for handlers that make their own durable calls (ctx.step, page fan-out, ctx.wait_for): code between those calls must be deterministic.
  • Optional cache_material(upstream, node, scope, info) adds key material (inputs content hashes, engine_versions, model_snapshots, parse_profile).

JSON Schemas of Classification, DocumentSegment, ScopeSpec and NodeOutcome (versioned): schemas/interpreter/ (python -m extractit.workflows.contracts --out schemas/interpreter).

Settings

Variable Default Meaning
DX_PARSE_PAGE_CONCURRENCY 8 Page tasks in flight per file
DX_PARSE_PAGES_PER_TASK 1 Pages per task
DX_WORKFLOW_MAX_FANOUT 8 Instances in flight per wave (per run: options.max_fanout)
DX_INTERPRETER_{CPU,LLM,IO}_CONCURRENCY 4/16/16 Child workflows of each class per worker
DX_INTERPRETER_PAGE_QUEUE_CONCURRENCY 16 Page tasks per worker
DX_INTERPRETER_QUEUE_POLLING_SEC / DX_INTERPRETER_RESULT_POLLING_SEC 0.1 / 0.05 Polling of the interpreter's queues and waits
DX_REVIEW_TIMEOUT_HOURS 168 Review timeout when the policy sets none

Extension point for usage metering

interpreter.activity_closed: on_activity_closed(hook) registers hook(session, event), called once per finished (node, scope) — succeeded, failed or reused from the cache — inside the transaction that closes its node_run, with tenant, project, run, node, scope, status, cache_hit, usage (model calls, tokens, cost, pages) and duration. T51 (usage, credits, budgets) plugs in here; the interpreter knows nothing about billing.