Skip to content

Workflow interpreter on the durable runtime

How T26 executes a workflow version on DBOS (T14). User-facing contract and routes: workflows/interpreter.md.

Layers

POST /v1/runs ──► InterpreterRunner.create ──► RunService.submit(kind="workflow")   (API process)
                    resolve alias once, pin version, packet (T17)

dx_run (T14, tier queue) ──► interpret(ctx)                                          (worker)
    load plan (immutable version, compile_plan of T25)
    loop: core.next_ready(plan, state)
        skip requests        ──► node_runs skipped (+ node.skipped)
        inline nodes         ──► begin ► run handler in the workflow ► finish
        cpu / llm / io nodes ──► begin ► child dx_wf_node on dx_wf_<class> ► finish
                                 (bounded window of max_fanout, awaited in order)
        core.apply(state, outcome)
    core.run_status ► result (run_results, kind workflow) ► dx_run sets the run status
  • Pure core (interpreter/core.py): next_ready(plan, state) and apply(state, request, outcome); no clock, randomness, I/O, database, HTTP or gateway (import-linter contract interpreter-core-is-pure). Requests come in (node_id, scope_key) order; any completion order of independent steps reaches the same final state (Hypothesis test).
  • Bookkeeping as checkpointed steps: begin (node_runs → running, attempt + 1, cache lookup, node.started) and finish (final status, node.completed/node.failed, interpreter.activity_closed) are DBOS steps of the parent; events go to T14's outbox in the same transaction as the state change, so sequence has no gaps.
  • Children: one dx_wf_node(run, node, scope, attempt) per instance on the queue of its resource_class, with the run's priority; the child id carries the attempt. A plain handler is one T14 step (execute_step, 3 attempts with exponential backoff for cpu/llm, 1 for io); an orchestrating handler (parse, review) makes its own durable calls.
  • Page fan-out: the parse child starts one dx_wf_pages per task on queue dx_wf_pages, at most DX_PARSE_PAGE_CONCURRENCY in flight (fanout.bounded_map), each a step whose raw pages are a content-addressed blob. Waits use DBOS wait_first with a short polling interval (DBOS's get_result polls once a second, too slow for hundreds of 200 ms tasks).

Crash, resume, rerun

  • SIGKILL: DBOS recovers the parent with the same executor id and replays its checkpoints: begin/finish are not repeated and children re-attach by id; a finished child returns its recorded result. Only work in flight runs again (at most the window).
  • Resume (:resume, T17 → RunService.resume): runs of kind workflow are registered with set_resume_from_start, so the failed run is rewound to its first step. The interpreter re-plans; begin finds the succeeded node_run of every finished node and reuses its outputs without starting a child; the failed node gets attempt + 1 (a new child id) and its descendants run. The run id does not change. (A rewind to the failed DBOS step would not do: the failure happened in a child, whose id DBOS would re-attach to.)
  • Rerun (:rerun): a new run with parent_run_id and trigger.kind = rerun. Not a DBOS fork — a fork reuses the original workflow arguments, the pinned version among them. The savings come from the per-node cache: cache_key = sha256(JCS{node_type, config_hash, input content hashes, scope pages, schema_hashes, model snapshot, engine_versions, parse profile}); a succeeded node_run of the project with the same key is reused (cache_hit).

Human review

The review node waits with T14's ctx.wait_for: the run records waiting_review (event run.waiting_review) and cancels its own workflow, which frees the queue slot and the thread (a recv would hold both for days); RunService.deliver — called by :complete and, later, T33's inbox — sends the payload and resumes it. Nothing else is in flight while a review waits: a wave runs its inline nodes before starting children, and children are awaited before the next wave. Timeouts are enforced by T14's reconcile (the node fails with REVIEW_TIMEOUT).

Cancellation

RunService.cancel marks the run and cancels its workflow with its children (cancel_children). Handlers check ctx.cancelled() between pages and model calls; the parent marks the unfinished node_runs cancelled and the nodes never reached skipped (skip_reason = cancelled), and T14 emits run.cancelled once.

Extension point interpreter.activity_closed

Declared for T51 (usage, credits, plans and budgets; ee/): on_activity_closed(hook); called once per closed billable activity — a (node, scope) that finished, also when reused from the cache — inside the transaction that closes the node_run, with run, node, scope, status, usage (model calls, tokens, cost, pages) and duration. Exactly once also across crashes: the closing step is checkpointed. No billing logic lives in the interpreter.

Additive changes to the runtime (T14)

  • dbos_app: start_child/child_result (child workflows by explicit id, fast result wait), register_plain_queue (the interpreter's class queues), EngineClient.cancel(..., cancel_children=).
  • handlers.set_resume_from_start(kind) and its use in RunService.resume.
  • RunService.submit(parent_run_id=) and cancel propagating to child workflows.

Deviations from the ticket

  • The parent is T14's synchronous dx_run workflow, not an async one: the long waits (reviews) leave the slot through T14's park mechanism, which already survives restarts.
  • Node artifacts reuse T17's run_artifacts (metadata names node and scope) instead of a second table; the migration 0007_t26 creates node_runs only.
  • The review wait runs in the parent (T14 wait_for), not in a child started with start_workflow: same effect (no slot held), one wait mechanism in the codebase.