Skip to content

Ingest and one-shot runs

Packets, uploads and the one-shot parse / extract runs of the server (T17). Every route follows the API conventions: RFC 9457 problems, Idempotency-Key on POST, keyset pagination, rate classes and audit events. Resources of another tenant (or project) are always 404.

Flows

Multipart packet

curl -sS -X POST "$DX/v1/packets" -H "Authorization: Bearer $KEY" \
  -F files=@invoice.pdf -F files=@delivery-note.pdf \
  -F 'metadata={"customer":"ACME"}' -F notes="March batch"

201 with the packet: {id: pkt_…, status: ready, file_count, content_hash, files[{id: file_…, name, sha256, size, media_type, page_count, source, warnings}]}. 1 to max_files_per_packet files; duplicate names become a.pdf, a (2).pdf. metadata is a JSON object of at most 64 KiB stored with the packet; it is never injected into a prompt (citable metadata arrives with T24). GET /v1/packets/{id} lists the files; DELETE hides the packet (certified deletion is T50).

URL import

curl -sS -X POST "$DX/v1/packets" -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["https://files.example.com/a.pdf", "https://files.example.com/b.pdf"]}'

202 with the packet in importing; an import run downloads every URL and the packet becomes ready or, if any URL fails, failed with errors[{url_hash, code}] and no files (atomic). POST /v1/parse and POST /v1/extract also accept {"url": …} (downloaded inside the request).

SSRF defence: only https, port 443 (DX_IMPORT_ALLOWED_PORTS), no credentials in the URL; the host is resolved with getaddrinfo and every address must be public (no private, loopback, link-local, multicast, reserved, unspecified, CGNAT 100.64.0.0/10, metadata 169.254.169.254, ULA fc00::/7, IPv4-mapped IPv6; IPs written in decimal, octal, hex or short form are decoded first). The connection is pinned to the validated IP (URL with the IP, Host header and the TLS sni_hostname, so the certificate is still verified), trust_env=False, at most 3 redirects — each one validated again —, 5 s to connect and DX_IMPORT_TIMEOUT_S (30 s) in total, and the body is counted while streaming. Logs and errors hold only scheme, host and a hash of the URL; the URL list travels to the worker as an encrypted blob. The tested forbidden targets live in tests/fixtures/ingest/ssrf_targets.json (43 entries).

Direct (presigned) upload

curl -sS -X POST "$DX/v1/uploads" -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"filename": "scan.pdf", "size": 63000000, "checksum_sha256": "…"}'
# → {id: upl_…, upload_url, method: PUT, headers, expires_at}
curl -sS -X PUT "$UPLOAD_URL" -H 'Content-Type: application/pdf' --data-binary @scan.pdf
curl -sS -X POST "$DX/v1/uploads/upl_…/complete" -H "Authorization: Bearer $KEY"

With S3 the upload_url is a presigned PUT (15 min) to t/{tenant_id}/incoming/{upload_id}; with local storage it is PUT /v1/uploads/{id}/content?token=… (send the API key too), a token base64url(upload_id|exp) signed with HMAC-SHA256 for 15 minutes. The staging area holds the bytes in clear until complete: use bucket encryption and a lifecycle rule that expires t/*/incoming/ after 24 h. complete reads the staged object in streaming, checks the declared size (upload-size-mismatch), checksum_sha256 (upload-checksum-mismatch), type and limits, encrypts it (DXE1) into the content-addressed store, creates the file (a one-file packet) and deletes the staged object; repeating it returns the same file_id. purge_stale_uploads (every 15 minutes) expires sessions not completed within 24 h. A single S3 PUT accepts up to 5 GiB, above every plan limit.

One-shot runs

# parse
curl -sS -X POST "$DX/v1/parse" -H "Authorization: Bearer $KEY" -F file=@invoice.pdf
# extract with an inline schema and rules, waiting up to 30 s
curl -sS -X POST "$DX/v1/extract?wait=30" -H "Authorization: Bearer $KEY" \
  -F file=@invoice.pdf -F "schema=$(cat invoice.schema.json)" -F rules=@rules.json
# or JSON with a packet
curl -sS -X POST "$DX/v1/extract" -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"packet_id": "pkt_…", "schema": {…}, "tier": "standard", "dedup": "reuse"}'

202 with the Run and Location: /v1/runs/{id}. Fields: file (multipart) or packet_id or url; schema (JSON Schema with x-dx, required for extract), rules (JSON rules of T09: a list or {rules: […]}), tier (fast|standard|agentic), model, priority (0-9), dedup. The schema and the rules are validated before anything is created: 400 with a JSON Pointer (/schema/…, /rules/0/check/equals). A packet with more than one file is 422 packet-multi-file-unsupported (T24 treats a packet as one document).

The extract run executes five durable steps — prepare (load, convert with T16 when the file is not a PDF; the derivative is a file with source.kind = derived), parse (DocIR stored gzip-compressed, page PNGs), extract (model call and citations, T06), verify_score (normalization, verification, rules, confidence and assembly — finalize of T08) and persist (result blob + summary). parse runs prepare, parse, persist. Completed steps are checkpointed: :resume re-executes only the unfinished ones.

?wait=N (N ≤ 60) polls every 250 ms with short transactions after the request transaction is committed: a finished run answers 200 ({run, result} for extract, the Run for parse), an expired wait 202 with the current state. At most 50 waits per project run at once; beyond that the wait is ignored and the answer is 202.

Runs and results

Route
GET /v1/runs?status=&kind=&created_from= keyset listing
GET /v1/runs/{id}?wait=N {id, kind, status, packet_id, steps[{key, status, attempt, duration_ms, cost_usd}], usage, error, timings, links}
POST /v1/runs/{id}:cancel queued → never starts; running → stops at the next step boundary; cancelled → 200 (idempotent); other terminal → 409
POST /v1/runs/{id}:resume failed/cancelled → 202, from the first unfinished step; anything else → 409 run-not-resumable
GET /v1/runs/{id}/result?format=full\|data\|fields\|flat full = {run_id, kind, packet_id, file_id, document{pdoc_id, pages[{page, width_px, height_px, dpi, route}]}, output} where output is the output contract of T08 (schemas/output-1.0.json)
GET /v1/runs/{id}/evidence?path=total the field and its evidence
GET /v1/runs/{id}/document?format=docir\|markdown\|blocks DocIR of T02, anchored markdown (one anchor per line), layout blocks
GET /v1/runs/{id}/pages/{n}.png ETag = sha256, Cache-Control: private, max-age=3600, If-None-Match → 304

A result above 5 MiB answers {result_url, expires_at, size}: GET /v1/runs/{id}/result/download?token=… with the API key, valid for 15 minutes (then 403). Blobs are encrypted with the tenant's DEK, so even on S3 the URL goes through the API (a presigned object URL would only return ciphertext).

Limits per plan

From project_limits (T12; projects without a row use the plan of DX_PLAN):

Limit Default Problem
max_file_mb 25 (MiB) 413 payload-too-large with max_bytes (26214400)
max_files_per_packet 100 413 too-many-files
max_pages_per_file unlimited in default (2000 if unset) 413 too-many-pages
max_concurrent_runs, max_queued_runs runtime defaults (10, 10000) queue tier and 429 quota-exceeded

Content-Length is checked before the body is read, and bytes are counted while copying, so a lying client is cut off at the limit. project_limits is also the RuntimeLimits provider of the runtime (T14) for runs submitted by the API.

Deduplication

content_hash = sha256(sorted sha256 of the files); the run fingerprint adds the run kind, schema hash, rules hash, tier and model. The blob is always shared inside the tenant (one object per sha256, never across tenants).

dedup Same document, schema, tier and policy already succeeded
reprocess (default) new run
reuse 200 with the previous Run and Deduplicated: true
reject 409 duplicate-document with existing_run_id

Problems

Code Status When
payload-too-large 413 a file above max_file_mb (max_bytes, file)
too-many-files 413 more than max_files_per_packet
too-many-pages 413 a PDF above max_pages_per_file
unsupported-media-type 415 the content is not a supported format (ZIP without Office structure, executables…); a mismatching extension with a supported content only warns extension_mismatch
pdf-encrypted 422 password-protected PDF (file, errors[])
invalid-document 422 corrupt PDF
url-not-allowed 422 SSRF policy
import-timeout, import-failed 422 download too slow / HTTP error
upload-token-expired 403 upload token or result URL forged, foreign or expired
upload-checksum-mismatch, upload-size-mismatch, upload-not-found 422 complete checks
duplicate-document 409 dedup=reject
packet-multi-file-unsupported 422 /extract with several files
packet-not-ready 409 packet still importing or failed
run-not-resumable 409 :resume of a run that is not failed or cancelled
run-not-finished 409 result or document of a run that has none yet