Skip to content

Parsing digital PDFs (tier Flash)

extractit.parse turns a digital PDF into a DocIR (words with exact boxes, lines, typed blocks in reading order, tables with cells, stable anchors) and renders it as anchored markdown. No language model is called: Heron (layout) and TableFormer (tables) run locally on CPU, words come from the PDF text layer.

from pathlib import Path

from extractit.parse import FileParseCache, ParseProfile, parse, to_anchored_markdown

doc = parse(Path("invoice.pdf"), ParseProfile(), cache=FileParseCache())
am = to_anchored_markdown(doc)  # AnchoredMarkdown: text, anchors, page_map
print(am.text)
extractit parse invoice.pdf -o out/            # docir.json, document.md, anchored.json, pages/p001.png
extractit parse invoice.pdf --format blocks    # only the blocks, as JSON, on stdout
extractit parse invoice.pdf --layout heuristic # no models: XY-cut order, every block a paragraph
extractit models pull parse                    # download the pinned weights (~0.5 GB, once)

Status lines (cache: hit (3 ms), page and word counts, warnings) go to stderr, so --format output on stdout stays clean. Exit code 0 on success (also for a page without a text layer, which only warns needs_ocr), 2 for an unreadable PDF (corrupt, encrypted, empty, io: ParseError, no traceback, no document text).

Pipeline

  1. render.page_geometries (pypdfium2): displayed size, /Rotate and CropBox of every page. The displayed page is the space of all normalized boxes: origin top-left, [0, 1].
  2. DoclingEngine.analyze (the only module importing Docling): DocumentConverter with OCR off, TableFormer on, generate_parsed_pages. Words and lines are docling-parse's; regions are Heron's clusters; the reading order is the position of the Docling document item that best overlaps (IoU ≥ 0.5) each cluster; tables are TableFormer's cells with spans. Pages go through the models in batches (page_batch, default 10) to bound memory.
  3. build.build_document (pure Python over RawPage): each word goes to the region that holds its center (the highest score wins) and to its docling-parse line; lines are cut where they cross a region border; blocks follow the engine's order (XY-cut for regions it did not order; headers first, footers last); lines and blocks are numbered from 1 per page in reading order; headings get level, parent_id and the outline.
  4. markdown.to_anchored_markdown: the single renderer of anchored text.

Without weights, or with layout='heuristic', docling-parse runs alone (no PyTorch) and xycut.py orders the page: every block is a paragraph, Block.confidences.layout is None and the document carries the warning layout_model_unavailable. A page without text layer has route = none, no words, and the warning needs_ocr (the OCR is T05).

Anchors and ids

Anchor Names Numbering
p{n}_l{m} a line from 1, per page, in reading order
p{n}_b{m} a block from 1, per page, in reading order
p{n}_t{k} a table from 1, per page
p{n}_t{k}_r{j} a table row (only in anchored text and evidence) j is the 0-based row of TableCell.row

Anchors and element ids (el_…, tbl_…) derive from the content: parsing the same file with the same profile gives the same anchors and ids. ParsedDocument.id derives from parse_cache_key.

to_anchored_markdown (the only renderer)

to_anchored_markdown(doc, granularity='line'|'block'|'both', include_furniture=True) -> AnchoredMarkdown is the only implementation of anchored text; the extraction harness (T06), the pipeline (T08) and agent mode (T10) consume its result and never render text themselves. The default granularity is line until T08 applies ADR-019. Page headers and footers are included by default because they may hold data of the issuer.

line (one anchor per line):

<!-- page 1 -->

## <a id="p1_l1"></a>Sales by region

<a id="p1_t1"></a>
<table>
<tr data-r="p1_t1_r0"><th colspan="2"><a id="p1_l2"></a>Quarter 1</th><th rowspan="2"><a id="p1_l3"></a>Total</th></tr>
<tr data-r="p1_t1_r1"><th><a id="p1_l4"></a>Units</th><th><a id="p1_l5"></a>Revenue</th></tr>
</table>

<a id="p1_l6"></a>The supplier delivers the goods listed below
<a id="p1_l7"></a>to the address shown in the header.

block (one anchor per block, no line anchors; the heading prefix stays on the same line):

## <a id="p1_b1"></a>Sales by region

<a id="p1_b3"></a>The supplier delivers the goods listed below
to the address shown in the header.

both (hierarchical: the block anchor, then the anchors of its lines):

<a id="p1_b3"></a><a id="p1_l6"></a>The supplier delivers the goods listed below
<a id="p1_l7"></a>to the address shown in the header.

Rules that hold for every granularity:

  • AnchorRef.char_start:char_end is the exact span an anchor names: the escaped text of a line, the text of a block from its first line, the <table>…</table> element of a table, the <tr>…</tr> element of a row.
  • page_map tiles text without gaps or overlaps; each page starts with <!-- page N -->.
  • Escaping. Document text is escaped (& → &amp;, < → &lt;, > → &gt;), so a PDF that contains <a id="p1_l1"></a> produces &lt;a id="p1_l1"&gt;&lt;/a&gt; and cannot forge an anchor: the only <a id=…> tags and <tr data-r=…> rows in the output are the renderer's. Consumers decode entities (html.unescape) to compare with Line.text.
  • Headings: # for title, ## for a level-1 section_header, ### for level 2…; a list item gets - unless its text already starts with a bullet or a number. Hyphens at line end and ligatures are kept as they are.
  • A table cell holds the anchored lines of its words; every line has exactly one anchor, even if the engine put a line outside the cells (it is written after the table).
  • Rendering is deterministic and has no dependence on the engine: it is a function of the ParsedDocument.

Profile, cache and models

ParseProfile(layout, tableformer_mode, dpi, page_batch): everything that changes the result is hashed (page_batch is not). parse_cache_key = sha256(file_sha256 ‖ profile_hash ‖ engine_versions_hash), where the engine versions are those of docling, docling-parse, docling-ibm-models, pypdfium2, the pinned layout and TableFormer revisions (parse/models.lock.toml), the layout mode, the TableFormer mode and DOCIR_VERSION. Computing the key imports none of them, so a cache hit does not load Docling or PyTorch (provenance.cache_hit = true). FileParseCache stores $DX_HOME/cache/parse/<key>.json.gz and, for the CLI, <key>/p001.png; a corrupt entry behaves as a miss.

The CLI imports each subcommand lazily (cli/main.py), so a cached extractit parse loads only the parse path: about 0.13 s of wall time for the whole process in a runtime install (uv sync --no-dev, Apple M5), of which the cache hit itself is 2 to 5 ms. A development environment is slower (about 0.5 s) because the dev group pulls rfc3987-syntax through cyclonedx-bom, and jsonschema then builds its grammar at import.

Weights: $DX_MODELS_DIR or ~/.extractit/models; extractit models pull parse downloads the revisions pinned in models.lock.toml and records the sha256 of every file in parse-manifest.json (for offline mirrors); extractit models status parse checks them without network. DX_PARSE_THREADS sets the model threads (default min(4, cpus)).

Known behavior of the models

  • Heron labels the headings of the synthetic fixtures section_header (not title) and reads Issue date: … lines as key_value. build derives level from numbering or font size, so the largest heading is level 1.
  • TableFormer can put a row-header cell at the end of its row's token sequence (the grid says col=3 for a cell at the left); tables.normalized_cells re-seats the single-column cells of a row in x order.
  • TableCell.bbox is the box of the cell's text (cell matching), not the ruled grid cell.
  • T19 refines every table after the DocIR is built (words reassigned to cells, headers, confidence, borderless tables, logical tables across pages): see Tables and logical tables.
  • Only Block.confidences.layout is filled here; per-page confidence and the parse confidence belong to T05.

Performance (hypothesis H2)

bench/parse_timing.py over 30 PDFs (73 pages) at 150 DPI and 4 threads, CPU only: median page 0.29 s (p95 0.55 s), peak RSS 2.3 GB, 6 s to load the models once. Full report with hardware: bench/reports/parse-timing.md.

Attributions

Docling (MIT), docling-parse (MIT), docling-ibm-models (MIT) and the layout model Heron (docling-project/docling-layout-heron, Apache-2.0) are IBM / Docling Project work. The TableFormer weights (docling-project/docling-models) are licensed CDLA-Permissive-2.0 and need this attribution (also in NOTICE). pypdfium2 (Apache-2.0 OR BSD-3-Clause) renders the pages. PyMuPDF and other copyleft PDF libraries are not used.