Skip to content

Route C: VLM per page aligned to words (tier Standard)

extractit parse --tier standard adds a third way of reading a page. Route A reads the PDF text layer, route B reads the page with OCR; route C sends the page image to a document VLM through the model gateway and aligns its text to the words of route A or B. The VLM gives the best reading order, tables and the text of hard scans; the words of A/B give the geometry. The VLM is never asked for boxes: every line keeps the boxes of its anchor words, so citations stay word-level.

extractit parse invoice.pdf --tier standard                      # the complex pages (T21 router)
extractit parse scan.pdf --tier standard --vlm-model gemini-38-flash
extractit parse report.pdf --tier standard --vlm-pages 1,3-4     # only these pages call the VLM
extractit parse invoice.pdf --tier standard --prompt-hint "Factura española, importes con coma decimal"

Since T21 the tier standard goes through the page router: only pages with a hard flag (photo, low DPI, script without OCR model, low OCR confidence) or complexity >= 0.40 call the VLM; --vlm-pages forces route C on the listed pages and keeps the others on A/B.

Status lines go to stderr (vlm: page 1 ok (codex-gpt56-sol) precision 1.00 recall 1.00, or vlm: page 2 failed: truncated). Pages outside --vlm-pages make no model call.

Engines (engines.toml)

The engine is configuration, never code: --vlm-model <id> or DX_VLM_MODEL=<id> picks an entry of src/extractit/parse/vlm/engines.toml (bench/vlm/engines.toml is the same file). DX_VLM_ENGINES points to another catalogue.

Entry Model Needs Notes
codex-gpt56-sol (default) codex:gpt-5.6-sol codex login (ChatGPT plan) measured on align-v0
gemini-38-flash gemini:gemini-3.8-flash DX_GEMINI_API_KEY estimated 4.73 USD / 1,000 pages
paddleocr-vl compat:PaddlePaddle/PaddleOCR-VL DX_COMPAT_BASE_URL (vLLM, GPU) fixed task prompt OCR:, plain text
qwen3-vl-8b compat:Qwen/Qwen3-VL-8B-Instruct DX_COMPAT_BASE_URL (vLLM, GPU) JSON by guided decoding
openai-gpt6-luna openai:gpt-6-luna DX_OPENAI_API_KEY estimated 0.63 USD / 1,000 pages
anthropic-haiku55 anthropic:claude-haiku-5-5 DX_ANTHROPIC_API_KEY role crops (re-reads of T21/T56)

Each entry declares model (a pinned provider:snapshot route, never latest), role (page or crops), dpi and max_px of the page image, max_tokens per page, concurrency, free_prompt, json_mode, air_gapped, regions (where the provider processes the page) and cost_per_1k_pages_usd with its basis (estimated, measured, subscription, self-hosted). EngineCapabilities of every engine says word_boxes=false (geometry comes from the anchor words), whether it is air-gapped, takes a free prompt and what it costs.

PaddleOCR-VL or Qwen-VL on vLLM. Serve the model with vLLM (vllm serve PaddlePaddle/PaddleOCR-VL) and set DX_COMPAT_BASE_URL=http://<host>:8000/v1 (and DX_COMPAT_API_KEY if the server wants one). PaddleOCR-VL has no JSON mode and a fixed task prompt: the engine sends the image with OCR: and wraps the plain-text answer into the page schema. GPU deployment and mirrored weights are part of the air-gapped profile (T54).

Policy. The model policy of the gateway (DX_MODEL_POLICY: air_gapped, eu_only, allowed_providers…) is applied before any call; an engine it rejects leaves the page with its A/B result and the warning no_eligible_engine.

Concurrency and cost. Inside one document at most DX_VLM_CONCURRENCY pages (4, capped by the entry) are read at once. A PageBudget (vlm_max_usd_per_page, 0.05 USD) refuses a call that would exceed the cap (reread_budget_exhausted). Every call is recorded in the usage ledger (tokens, cost, latency) and summed in provenance.cost; provenance.model_calls lists the calls and provenance.engines the engine with its snapshot.

What the alignment produces

The VLM markdown is split into blocks (paragraphs, headings, list items, tables with their cells flattened by rows, formulas), paired with the layout blocks of the page by a monotone dynamic programme (cost 1 − Jaccard of word 4-grams plus a length prior, skip 0.8; pairs above 0.7 are not trusted), then aligned word by word with rapidfuzz Levenshtein opcodes over normalized tokens (NFKC, ligatures, quotes, dashes, case, end-of-line hyphens; digits compared as digits). Tokens the monotone pass leaves unpaired are matched exactly afterwards, so a VLM that reads a table row by row while the layout read it by columns still aligns.

Every aligned line keeps its anchor words (a word never moves to another line) and gets, in Line.alignment (DocIR 1.2):

Field Meaning
provenance how the text is grounded (ladder below)
alignment_score share of the line's VLM characters aligned to an anchor word
flags divergences (below)
polygon envelope of the aligned words of the line
spans text[start:end] → word (equal, replace, conflict, insert, kept)
alternatives the VLM reading of a word whose anchor text was kept

Lines and blocks are rewritten in the VLM reading order and their anchors renumbered (p1_l1…, p1_b1…) deterministically. PageQuality gets alignment_precision, alignment_recall and alignment_score.

Provenance ladder

  1. exact_text_layer: a digital page; the text is the text layer (the VLM gives order and structure; a CER above 0.10 against the text layer adds text_layer_disagreement).
  2. ocr_aligned: VLM text aligned to OCR words.
  3. vlm_block_only: VLM text no word explains; it keeps only the box of its layout block.
  4. interpolated: VLM text with a box interpolated from its neighbours.
  5. none: no geometry at all.

Flags

Flag When
interpolated the VLM wrote tokens no anchor word explains (table markup and formulas excepted)
possible_hallucination 8 or more interpolated tokens in a row
omission 3 or more anchor words (or 15 characters) in a row that the VLM left out
numeric_conflict a substituted token whose digits differ: the anchor text is kept, the VLM reading goes to alternatives
text_layer_disagreement digital page, VLM text vs text layer CER > 0.10

Citations

ParsedDocument.words_for_span(element_id, (start, end)) turns characters of a line (or block) text into anchor words. The citation verifier (T07) uses the alignment spans of aligned lines, so a value read from VLM text is cited on the words that hold it.

When a page fails

Guards check every answer before aligning: repetition_loop (zlib ratio < 0.2 over 400 characters, or a 4-gram repeated 8 times), truncated (max_tokens reached), refusal, empty, length_ratio (VLM/anchor characters outside [0.5, 2.0]), plus no_eligible_engine (policy), reread_budget_exhausted and engine_error. The page keeps its route A/B result, gets the warning vlm_failed and is listed in quality.pages_flagged; retries and engine fallback are the router's job (T21). A failed document is not cached, so the next run tries again.

Prompt hint

--prompt-hint (≤ 500 characters, else invalid-prompt-hint, HTTP 422) describes the document. Control characters and < > & are escaped and the hint travels in a <hint>…</hint> block that the versioned system prompt (PROMPT_VERSION) declares as data, never instructions; the page image is declared data too. The hint is part of parse_cache_key; only its length and sha256 are logged or stored. Engines with a fixed task prompt (free_prompt = false) do not receive it and the document gets the warning prompt_hint_ignored.

Crop primitives (T21, T56)

render_crop(page_image, bbox, scale=2.0, max_px=2048) cuts a box and scales it (400 DPI from a 200 DPI render), lowering the scale so no side exceeds max_px. CropReader(gateway).read(crop, Schema, purpose) returns a validated Pydantic object (the gateway repairs an invalid answer at most twice; purpose goes to the ledger). Both declare the image as data.

Measuring engines

uv run python bench/vlm/eval_align_v0.py --write-report    # bench/reports/align-v0.md
uv run python bench/vlm/run_matrix.py --write-report       # bench/reports/vlm-engines.md
DX_LIVE=1 uv run python bench/vlm/record_align_v0.py --engine <id>   # record a new engine (real calls)

bench/datasets/align-v0 holds 40 one-page invoices from the generator of T10 (20 digital, 20 scanned at 200 DPI with light degradation) with ground-truth values and boxes, and the frozen inputs of the aligner (frozen.json): the page image the VLM read and the DocIR of route A/B, each with its sha256. PDFium renders and extracts text differently on macOS and Linux, so the evaluation and the cassettes (tests/cassettes/vlm/) read these frozen inputs; the cassette key is the hash of the request, which contains the frozen image and never the local render. DX_VLM_IMAGES=<dir> makes route C read frozen page images (<dir>/<pdf sha256>/p001.jpg) in any parse, which is how the CLI tests replay cassettes on every platform.