Skip to content

Complete tables in the extraction

The rows of a long table never go through the output of a model. For every field with x-dx.table (an array of objects) the table handler (extractit.extract.tables) takes the logical table that T19 built (one table, possibly stitched across pages), asks a model once per document how the columns map to the properties, and then materializes every row in code, cell by cell, with the typed normalizers of T07 and an evidence that points at the cell. At the end every physical row is accounted for.

from extractit import extract

out = extract("statement-50p.pdf", "statement.schema.json", model="codex")
out.table_stats["movements"]
# rows_physical=2400 rows_extracted=2399 rows_excluded={'total': 1} rows_merged=0
# completeness='complete' mode='mapped'
[v for v in out.validations if v.rule_id.startswith("implicit:row_count:")]  # status 'pass'

Flow

x-dx.table fields ──plan──▶ candidate tables (T19 logical tables, or a physical table alone)
                              │
              mapping ◀───────┤  1 call per document: header, 3 sample rows, properties → columns,
                              │  row rules, confidence
                              ▼
        reliable? ── no ──▶ fallback: rows in batches of 25 with row anchors
            │ yes
            ▼
   row rules (RE2) ──▶ exclude / merge ──▶ anomalous rows ──▶ batches of 20: action + cells
            │
            ▼
   materialization (streaming): value = normalize(text of the cell) + `table_cell` evidence
            │
            ▼
   completeness: extracted + excluded + merged = physical?  → `table_stats` and rule `row_count`
  • Planning (table_ops/plan.py). The handler (kind table, priority 10, entry point extractit.field_handlers) claims the arrays of objects with x-dx.table when the document has tables and a model is available; T23 (priority 5) only claims scalars, so a mixed schema is split between them. Properties that are themselves arrays or objects (nested tables) are not extracted and get a warning.
  • Mapping (mapping.py). One structured-output call (purpose = "extract", phase table_mapping, low-tier model, temperature 0). The model sees, per candidate table, its id, canonical header (header_paths of T19, so a two-level header reads Importe / Neto), number of rows, pages and three sample rows; plus the properties with their description, instructions and x-dx.table.hint. The document content is delimited as data. The answer {tables: [{field_path, logical_table_id, columns: {prop: index | null}, row_rules, confidence}]} is validated against its JSON Schema (schemas/table-mapping-response-1.0.json, and a per-request schema that pins the keys of columns), then against the candidates: indices in range, and two properties never share a column unless their types are compatible. A table can be mapped several times when it continues unstitched in another table (rows are numbered on).
  • Row rules (rowrules.py). A declarative DSL, no code execution: exclude_if and merge_into_previous_if, each with up to 6 conditions (regex, empty, equals, startswith, numeric_only, optionally negated) on a property or on the whole row (*). Patterns are RE2 (google-re2), at most 200 characters. A pattern that is too long or not valid RE2 raises invalid-row-rule; the mapping call is repeated once (the retry says why), and if it fails again the table is read by the fallback.
  • Materialization (materialize.py). Each value is the text of its cell, normalized with the T07 normalizers. The decimal separator of a number property is voted by the whole column (so 1.234,56 and 1,234.56 both give 1234.56, and an ambiguous 1.234 follows its column), unless the schema fixes it with x-dx.normalize.decimal_separator. A cell that cannot be normalized keeps value = null and its raw_text, is abstained, and raises the warning cell_not_normalizable (the first 20, then one summary): the table goes on. Rows are produced one at a time.
  • Evidence. Every cell gets one table_cell evidence: anchor is the cell anchor (p3_t1_r17_c4), table_ref = {table_id, logical_table_id, row, col} (physical row and column), words and word_range (only when the words are contiguous: a cell's words need not be) and one region per physical row part. A row cut by a page break therefore cites two regions on two pages. Verification is by construction (ground/cells.py): the value is read from the cited words, so it is verified; a cell with text but no words is unverified.
  • Anomalous rows (anomalies.py). A row is anomalous when a typed cell cannot be normalized, a cell spans several columns, fewer than 40 % of the mapped columns have text, or its key is empty while another cell has text. Only those rows go to the model, in batches of 20, with the header and the previous row. The answer is an action (keep, exclude, merge_into_previous) and, per property, which columns feed it: never a value. Values are rebuilt from the cells. More than 10 % anomalous rows raise many_anomalous_rows.
  • Fallback (fallback.py). A table without a detectable header, a mapping with confidence below 0.5, or a required property without a column is read in batches of 25 rows as HTML with a row anchor per <tr>. The model answers one object per anchor (data with values, or skip); every quoted value is checked against the cells of its own row (a value that is not there is dropped), and a physical row it did not answer is asked once more.
  • Completeness (completeness.py). table_stats[path] = {rows_physical, rows_extracted, rows_excluded{rule: n}, rows_merged, completeness, missing_rows, mode, tables, stitch_suspect}. rows_physical is n_rows of the logical table; complete means extracted + excluded + merged = physical. A stitch that T19 flagged suspect_table_stitch adds the warning stitch_suspect and leaves the cells in needs_review.

The implicit rule row_count

For x-dx.table.completeness = all_rows (the default for a table field) the handler declares the rule implicit:row_count:<field_id>: a row_count check with against = "table_stats" that the rules engine of T09 evaluates with the rest of the rules. It compares the rows that reached the output with rows_physical - rows_excluded - rows_merged: pass when they match, fail with faltan N filas when some are gone. best_effort declares no rule. (ExtractionView.table_stats carries the numbers to the engine.)

What a table needs to be extracted well

  • A header that T19 detected (or the fallback reads it without one).
  • A schema whose properties have descriptions in the document's words; x-dx.table.hint says what to exclude ("excluye subtotales"); x-dx.table.row_key names the property that identifies a row.
  • Marker rows ('Suma y sigue', 'Übertrag') are left out of the logical table by T19 before this handler sees it; totals and subtotals are rows and are excluded here with exclude_if.

Cost

One mapping call per document plus ceil(anomalous / 20) anomaly calls, whatever the number of rows (2 400 rows: 1 call and about 100 output tokens when nothing is anomalous). The fallback costs one call per 25 rows and its output grows with the rows: it is the exit for tables the mapping cannot describe, not the normal path. Calls are recorded in the usage ledger with purpose = "extract" and the phases table_mapping, table_anomalies and table_fallback.

Security

Table text is data: it is delimited, lookalike delimiters are broken, HTML cells are escaped in the fallback, every answer is validated against its JSON Schema, row rules run no code and use a linear-time engine, and neither warnings nor logs contain cell content.

Evaluation

bench/tables_eval.py runs long-tables-v0 (variants a–g, 1 to 50 pages) with bench/schemas/statement.schema.json over a DocIR frozen with the dataset, and writes bench/reports/ext-tables-v0.md. See that report for the numbers and for how the cassettes of tests/cassettes/table-mapping were recorded.