Complete tables in the extraction
The rows of a long table never go through the output of a model. For every field with x-dx.table
(an array of objects) the table handler (extractit.extract.tables) takes the logical table
that T19 built (one table, possibly stitched across pages), asks a model once per document how
the columns map to the properties, and then materializes every row in code, cell by cell, with
the typed normalizers of T07 and an evidence that points at the cell. At the end every physical row
is accounted for.
from extractit import extract
out = extract("statement-50p.pdf", "statement.schema.json", model="codex")
out.table_stats["movements"]
# rows_physical=2400 rows_extracted=2399 rows_excluded={'total': 1} rows_merged=0
# completeness='complete' mode='mapped'
[v for v in out.validations if v.rule_id.startswith("implicit:row_count:")] # status 'pass'
Flow
x-dx.table fields ──plan──▶ candidate tables (T19 logical tables, or a physical table alone)
│
mapping ◀───────┤ 1 call per document: header, 3 sample rows, properties → columns,
│ row rules, confidence
▼
reliable? ── no ──▶ fallback: rows in batches of 25 with row anchors
│ yes
▼
row rules (RE2) ──▶ exclude / merge ──▶ anomalous rows ──▶ batches of 20: action + cells
│
▼
materialization (streaming): value = normalize(text of the cell) + `table_cell` evidence
│
▼
completeness: extracted + excluded + merged = physical? → `table_stats` and rule `row_count`
- Planning (
table_ops/plan.py). The handler (kindtable, priority 10, entry pointextractit.field_handlers) claims the arrays of objects withx-dx.tablewhen the document has tables and a model is available; T23 (priority 5) only claims scalars, so a mixed schema is split between them. Properties that are themselves arrays or objects (nested tables) are not extracted and get a warning. - Mapping (
mapping.py). One structured-output call (purpose = "extract", phasetable_mapping, low-tier model, temperature 0). The model sees, per candidate table, its id, canonical header (header_pathsof T19, so a two-level header readsImporte / Neto), number of rows, pages and three sample rows; plus the properties with their description, instructions andx-dx.table.hint. The document content is delimited as data. The answer{tables: [{field_path, logical_table_id, columns: {prop: index | null}, row_rules, confidence}]}is validated against its JSON Schema (schemas/table-mapping-response-1.0.json, and a per-request schema that pins the keys ofcolumns), then against the candidates: indices in range, and two properties never share a column unless their types are compatible. A table can be mapped several times when it continues unstitched in another table (rows are numbered on). - Row rules (
rowrules.py). A declarative DSL, no code execution:exclude_ifandmerge_into_previous_if, each with up to 6 conditions (regex,empty,equals,startswith,numeric_only, optionally negated) on a property or on the whole row (*). Patterns are RE2 (google-re2), at most 200 characters. A pattern that is too long or not valid RE2 raisesinvalid-row-rule; the mapping call is repeated once (the retry says why), and if it fails again the table is read by the fallback. - Materialization (
materialize.py). Each value is the text of its cell, normalized with the T07 normalizers. The decimal separator of a number property is voted by the whole column (so1.234,56and1,234.56both give 1234.56, and an ambiguous1.234follows its column), unless the schema fixes it withx-dx.normalize.decimal_separator. A cell that cannot be normalized keepsvalue = nulland itsraw_text, isabstained, and raises the warningcell_not_normalizable(the first 20, then one summary): the table goes on. Rows are produced one at a time. - Evidence. Every cell gets one
table_cellevidence:anchoris the cell anchor (p3_t1_r17_c4),table_ref = {table_id, logical_table_id, row, col}(physical row and column),wordsandword_range(only when the words are contiguous: a cell's words need not be) and oneregionper physical row part. A row cut by a page break therefore cites two regions on two pages. Verification is by construction (ground/cells.py): the value is read from the cited words, so it isverified; a cell with text but no words isunverified. - Anomalous rows (
anomalies.py). A row is anomalous when a typed cell cannot be normalized, a cell spans several columns, fewer than 40 % of the mapped columns have text, or its key is empty while another cell has text. Only those rows go to the model, in batches of 20, with the header and the previous row. The answer is an action (keep,exclude,merge_into_previous) and, per property, which columns feed it: never a value. Values are rebuilt from the cells. More than 10 % anomalous rows raisemany_anomalous_rows. - Fallback (
fallback.py). A table without a detectable header, a mapping with confidence below 0.5, or a required property without a column is read in batches of 25 rows as HTML with a row anchor per<tr>. The model answers one object per anchor (datawith values, orskip); every quoted value is checked against the cells of its own row (a value that is not there is dropped), and a physical row it did not answer is asked once more. - Completeness (
completeness.py).table_stats[path] = {rows_physical, rows_extracted, rows_excluded{rule: n}, rows_merged, completeness, missing_rows, mode, tables, stitch_suspect}.rows_physicalisn_rowsof the logical table;completemeans extracted + excluded + merged = physical. A stitch that T19 flaggedsuspect_table_stitchadds the warningstitch_suspectand leaves the cells inneeds_review.
The implicit rule row_count
For x-dx.table.completeness = all_rows (the default for a table field) the handler declares the
rule implicit:row_count:<field_id>: a row_count check with against = "table_stats" that the
rules engine of T09 evaluates with the rest of the rules. It compares the rows that reached the
output with rows_physical - rows_excluded - rows_merged: pass when they match, fail with
faltan N filas when some are gone. best_effort declares no rule. (ExtractionView.table_stats
carries the numbers to the engine.)
What a table needs to be extracted well
- A header that T19 detected (or the fallback reads it without one).
- A schema whose properties have descriptions in the document's words;
x-dx.table.hintsays what to exclude ("excluye subtotales");x-dx.table.row_keynames the property that identifies a row. - Marker rows ('Suma y sigue', 'Übertrag') are left out of the logical table by T19 before this
handler sees it; totals and subtotals are rows and are excluded here with
exclude_if.
Cost
One mapping call per document plus ceil(anomalous / 20) anomaly calls, whatever the number of rows
(2 400 rows: 1 call and about 100 output tokens when nothing is anomalous). The fallback costs one
call per 25 rows and its output grows with the rows: it is the exit for tables the mapping cannot
describe, not the normal path. Calls are recorded in the usage ledger with purpose = "extract"
and the phases table_mapping, table_anomalies and table_fallback.
Security
Table text is data: it is delimited, lookalike delimiters are broken, HTML cells are escaped in the fallback, every answer is validated against its JSON Schema, row rules run no code and use a linear-time engine, and neither warnings nor logs contain cell content.
Evaluation
bench/tables_eval.py runs long-tables-v0 (variants a–g, 1 to 50 pages) with
bench/schemas/statement.schema.json over a DocIR frozen with the dataset, and writes
bench/reports/ext-tables-v0.md. See that report for the numbers and for how the cassettes of
tests/cassettes/table-mapping were recorded.