Core contracts
extractit.core holds the types every module shares, so that parsing, extraction,
verification, rules and evaluation can be built in parallel against fixtures. All models are
Pydantic v2, frozen and strict about unknown keys (extra="forbid"). Each contract is published
as a versioned JSON Schema in schemas/.
| Contract | Model | Version constant | Snapshot |
|---|---|---|---|
| Parsed document (DocIR) | core.docir.ParsedDocument |
DOCIR_VERSION |
schemas/docir-1.2.json (docir-1.0.json and docir-1.1.json stay for older readers) |
| Anchored text | core.anchored.AnchoredMarkdown |
ANCHORED_MARKDOWN_VERSION |
schemas/anchored-markdown-1.0.json |
| Extraction output | core.output.ExtractionOutput |
OUTPUT_SCHEMA_VERSION |
schemas/output-1.2.json (output-1.0.json and output-1.1.json stay for older readers) |
| Extraction-schema dialect | core/schema/metaschema.json |
EXTRACTION_SCHEMA_DIALECT_VERSION |
schemas/extraction-schema-meta-1.2.json (-1.1 stays) |
Versioning policy
- Versions are
MAJOR.MINORand live inextractit/core/version.py. - An additive change (new optional field, new enum value a reader may ignore) bumps
MINOR. An incompatible change (removed or renamed field, narrower type) bumps MAJOR
and needs a migration note in
CHANGELOG.md. - Until phase F1 closes, every contract stays at
1.x: only additive changes. - A version that is not in any release (no tag) absorbs further additive changes: regenerate its snapshot in place, without bumping another MINOR. Once it is published, additive = a new MINOR.
- When you bump, also add the new value to the matching
*Versionliteral so that documents written with an older minor keep loading, run the export and commit the new snapshot. Older snapshots stay inschemas/.
uv run python -m extractit.core.schemas export # write schemas/<name>-<version>.json
uv run python -m extractit.core.schemas check # exit 1 if a model changed without a bump
make contracts # runs scripts/contracts.d/*.sh (CI)
check exits 1 when the snapshot for the current version is missing or differs from what the
models generate. If only descriptions (docstrings) changed, it says so: re-export, no bump.
Identifiers and geometry
- Ids are TypeIDs:
<prefix>_<26 base32 chars>over a UUIDv7 (pdoc_,fld_,ev_,vr_,mc_,rule_,ext_,el_,tbl_…).new_id(prefix)is time-sortable;deterministic_id(prefix, key)hasheskey(sha256, first 16 bytes, UUIDv7 version/variant bits) and always returns the same valid id: the pipeline uses it so that two runs with the fake model produce identical output.id_to_uuid/id_from_uuidconvert foruuidcolumns. - Coordinates are normalized to the page as displayed (after
/Rotateand the CropBox), origin top-left,0 ≤ x0 < x1 ≤ 1,0 ≤ y0 < y1 ≤ 1(BBox). JSON output rounds boxes outwards to 4 decimals.to_pdf_space(bbox, page)/from_pdf_space(rect, page)convert to PDF user space (points, origin bottom-left) applying the inverse rotation first, then the CropBox.iouandunion_bboxare shared by the verifier and evaluation. - Canonical JSON:
serialize(obj)(sorted keys, compact, UTF-8) gives identical bytes for identical objects;canonical_hashis its sha256.
DocIR 1.2 (ParsedDocument)
ParsedDocument{docir_version, id: pdoc_…, sha256, parse_cache_key, profile{tier, profile_hash, options},
pages[], words[], lines[], blocks[], tables[], logical_tables[], outline[],
quality{parse_confidence, layout_confidence, pages_flagged, warnings},
provenance{engines[], model_calls[], started_at, finished_at, cache_hit}}
Page{number, width_pt, height_pt, rotation_applied, deskew_angle_deg, crop_box?, render?,
text_source, route, quality{text_layer_coverage, ocr_mean_conf, parse_confidence, …, complexity}}
Word{idx, page, text, bbox, polygon?, conf, source, line_idx, block_idx, font?}
Line{idx, id: el_…, anchor: p{n}_l{m}, page, bbox, text, word_range[start, end), block_idx}
Block{id: el_…, anchor: p{n}_b{m}, type, page, bbox, order, level, parent_id, text, markdown,
line_range, confidences{text, layout}, source{engine, model_version, route}}
TableStructure{id: tbl_…, anchor: p{n}_t{k}, page, block_idx, n_rows, n_cols, header_rows,
cells[{row, col, row_span, col_span, text, bbox, is_header, word_idxs, conf,
word_range?}],
column_x, method, confidence, continues?{from, to}, extra{}}
LogicalTable{id: ltbl_…, segments[{table_id, page, packet_page?, row_offset, header_repeated,
data_rows[]}], header, header_levels, n_rows, stitch_confidence, stitch_evidence[],
stitch_checks[{name, passed, detail}], split_rows[{row, parts[RowPart]}]}
RowPart{table_id, page, packet_page?, physical_row}; LogicalTable.row_ref(i) -> RowRef{row, parts}
DocIR 1.1 (T19, additive). TableCell.word_range is set only when the cell's words
(word_idxs, the word_ids of the T19 ticket) are contiguous; TableStructure.continues links a
table to the previous (from) and next (to) segment of its logical table and extra carries
open flags (nested, orphan_words, refined, structure_score); LogicalTable gains
header_levels, stitch_checks, split_rows, per-segment data_rows and packet_page, and
stitch_evidence gains geometry, numbering, column_types and override. Cell anchors
p{n}_t{k}_r{j}_c{c} (CELL_ANCHOR_PATTERN) appear as data-c in table HTML and resolve with
doc.resolve_anchor(anchor) → ResolvedAnchor{anchor, page, word_ids, bbox} (row anchors give
the union of the row's cells; an unknown anchor raises AnchorNotFound). They are not
AnchorRef entries: the anchored-text contract is unchanged. Documents written as 1.0 still load
(tests/fixtures/contracts/compat/docir-1.0-2p-120w.json). See Tables.
DocIR 1.2 (T20, additive) adds the VLM alignment of route vlm_page (see
VLM per page).
DocIR 1.2 also absorbs T21 (additive; 1.2 is not in any release yet, so its snapshot is regenerated in place, no 1.3): the page router and path D (see Page router):
TableCell.text_source (words | vlm_c | vlm_d | vote), TableCell.alternatives
([{source, text}], sources words | vlm_c | vlm_d | vlm_d2 | vlm_x) and TableCell.flags
(conflict | no_majority | numeric_conflict) on cells reconciled by path D;
PageQuality.cross_engine_agreement and DocumentQuality.cross_engine_agreement;
Provenance.routing (one PageRouting{page, path, route, engine, reason, expected_cost_usd,
complexity, signals, route_history[]} per page) and Provenance.fallbacks
(PageFallback{page, from, to, cause}). Every new field is omitted from the JSON when unset, so a
document parsed without the router serializes as before. The router's own contracts live in
schemas/router-{policy,routing,table-reading,parse-problem}-1.0.json
(scripts/contracts.d/router.sh).
ParseRoute is text_layer | ocr | vlm_page | vlm_agentic | table_specialist | external | none.
Pages are 1-based; words, lines and blocks are addressed by their position. Construction fails —
naming the culprit, e.g. words[7].line_idx — unless:
- every word belongs to exactly one line, line word ranges tile
wordsin order, and words agree with their line on page and block; idxfields are dense and equal to the position; page numbers are1..N;- anchors match their format, name the right page and are unique; element ids are unique;
Block.orderis a permutation of0..N-1; line/block/table/parent references exist (alsocontinues, logical-table segments, theirdata_rowsand split-row parts);- every box is a valid
BBoxin[0, 1].
Row anchors p{n}_t{k}_r{j} are only valid in anchored text and evidence (rows are not elements).
DocIR 1.2 (T24, additive). Page gains file_id, file_page and packet_page: in a packet (several
files seen as one document) the pages are numbered globally, anchors become p{packet_page}_l{n} and each
page records the file and the page inside it. They are null outside a packet. See
Field sources.
DocIR 1.2 (T20, additive). The VLM alignment of route vlm_page: Line.alignment (provenance
ladder, score, flags, polygon, character spans to words and alternative readings), Block.alignment,
the alignment metrics of PageQuality, Provenance.cost and ParsedDocument.words_for_span. Every
new field is omitted from the JSON when unset, so a document without VLM alignment serializes as
in 1.1. FieldEvidence.anchor also accepts a cell anchor (T22).
Anchored text 1.0 (AnchoredMarkdown)
The only bridge between the renderer (T04, sole producer) and the extraction harness (T06): T06 and T08/T10 never need the parser.
AnchoredMarkdown{anchored_markdown_version, renderer_version, granularity: line|block|both,
doc_sha256, text,
anchors[{anchor, kind: line|block|table|row, page, line?, block?, char_start, char_end}],
page_map[{page, char_start, char_end}]}
text[char_start:char_end] is the anchored content. Anchors are unique and well formed, their
kind and page match the anchor name, spans are non-empty, inside text and inside their page;
page_map lists pages in increasing order and tiles text with no gap or overlap. Helpers:
am.get(anchor), anchor in am, am.anchor_text(anchor), am.page_text(page).
Results and output 1.2
FieldCandidate{path, field_id, status_hint: extracted|not_found|abstained, value, raw_text, anchors, quote, model_call_id}— what the harness (T06) proposes; T07 verifies it.FieldEvidence{id: ev_…, kind, role, page, anchor, quote, char_span, word_range, words[{text, bbox, word_idx}], regions[{page, bbox, polygon}], table_ref, verification, match{method, score, normalized_value}, citation_corrected, produced_by}.ConfidenceScore{score, kind: calibrated_probability|heuristic, band, signals, calibrator?, threshold_policy_id?, reason?}.ValidationResult{id: vr_…, rule_id, rule_version, status: pass|fail|inconclusive|skipped, severity, blocking, detail, compared_values, evidence_ids, inconclusive_reason, model_call_id};inconclusive_reasonis only allowed whenstatusisinconclusive.Usage{model, model_snapshot, billing: metered|subscription|agent|none, input_tokens, output_tokens, cost_usd, duration_ms, steps[]}.ExtractedField{path, field_id, value, raw_text, status, confidence, evidence, alternatives, review_state}: every field is nullable;not_foundhasvalue = null,extracteddoes not.
ExtractionOutput{schema_version, extraction{mode: llm|agent, tier, model, agent},
document{sha256, pages, routes, pdoc_id, filename}, data, fields, document_confidence,
operator_confidence{parse, extract, validate}, requires_review, validations, usage,
reproducibility{schema_hash, prompt_hash, model_snapshot, engine_versions, parse_cache_key,
docir_version, tier, seeds, deterministic}, warnings, strategy, coverage} is what extractit FILE prints. Field paths
and evidence ids are unique and validations only cite evidence present in fields. The example
in tests/fixtures/contracts/valid/output-4-fields.json shows every part.
Output 1.1 (T23, additive). strategy is single_pass (default: one request over the whole
text) or long_document (location, windows and consolidation, see
Long documents); coverage{pages_covered, pages_total, pages_read,
failed_windows} is set only by the long-document strategy and pages_covered < pages_total means
a window failed. Outputs written as 1.0 still load.
Output 1.2 (T22 and T24, additive). table_stats{path: {rows_physical, rows_extracted,
rows_excluded, rows_merged, completeness, missing_rows, mode, tables, stitch_suspect,
anomalous_rows}} is set when an x-dx.table field was extracted (see
Complete tables); null otherwise. table_cell evidence may carry a
cell anchor (p3_t1_r17_c4) in anchor and one region per physical row part (a row cut by a
page break cites two pages). The rules contract 1.0 gains row_count with against: table_stats
and table (additive). FieldEvidence also gains lookup_ref{table_id, table_version, row_key,
matched_on, match_score} (kind='lookup_row'), derivation{expression, inputs} (kind='derived'),
metadata_path (kind='metadata') and file_id, file_page, packet_page (evidence on a packet);
ConfidenceSignals gains source_confidence (the match score times the weakest input, or the weakest
input of a derived field, which replaces the grounding base); a FieldCandidate may cite
metadata.<path> and notes.l<n> anchors. Outputs written as 1.0 and 1.1 still load. See
Field sources. output-1.1.json stays as it was published, for
older readers.
Output 1.2 (T39). ConfidenceScore.reason (optional): calibrator_stale when a
calibrator exists for the field but its model snapshot or prompt hash differ from the run's, so the
score stays heuristic. A calibrated_probability score carries
calibrator{id, version, scope{field_type, model_snapshot}, ece, n_samples, fitted_at}; see
Calibrated confidence. Outputs written as 1.0 or 1.1 still load.
Ports: ExtractionView and ValidatorPort
The pipeline (T08) and the rules engine (T09) only meet here. ExtractionView(fields, data=None)
(or ExtractionView.from_output(out)) is read-only: get/value/confidence/evidence_ids
by field_id or path, select("line_items[*].amount") and by_field_id(id) for every row,
row_count("line_items"), and data (rebuilt from the fields with data_from_fields if not
given). A rules engine implements:
class ValidatorPort(Protocol[RuleT]):
def evaluate(
self, rules: Sequence[RuleT], view: ExtractionView, /
) -> list[ValidationResult]: ...
Extraction-schema dialect 1.2
1.2 (T24, additive). x-dx.source is a string (the legacy values) or an object {kind, …}:
document, lookup, derived, metadata (and kinds registered by other packages). The legacy strings with
lookup, expression or metadata next to them normalize to the object (FieldDef.source_spec;
FieldDef.source keeps a flat label). See Field sources.
Restricted JSON Schema 2020-12 plus x-dx extensions. Allowed keywords on a field: type,
properties, required, items, enum, format, description, title, minimum, maximum,
x-dx. Types: string (formats date, date-time, email, uri, iban, bic, vat_id,
phone), number, integer, boolean, string enum, array of primitives or objects (an
array of enum strings is a multi_select), object. Nesting up to max_depth (4 by default;
x-dx.max_depth at the root or load_schema(..., max_depth=)).
x-dx key |
Meaning |
|---|---|
field_id |
Persistent identity (fld_ TypeID); survives renames. Assigned if absent. |
instructions |
Natural-language guidance, ≤ 2 000 characters. |
source |
document (default), lookup, lookup_if_missing, metadata, derived (executed from T24). |
evidence |
required (no verifiable evidence ⇒ abstained), preferred (default), none. |
normalize |
date_order, decimal_separator, thousands_separator, unit, case, strip, percent (points | fraction, since 1.1), enum_threshold (0–100, since 1.1). |
table |
Arrays of objects only: completeness (all_rows/best_effort), row_key, hint. |
lookup, expression |
Reference table / CEL expression for lookup/derived sources (T24). |
enum_descriptions |
Description per enum option. |
pii |
none, personal, special_category. |
Schema-level x-dx: schema_version, document_type, locale_hint, max_depth.
Every field is nullable: required means expected — its absence is a finding, never a
reason to invent a value.
Loading
from extractit.core.schema import load_schema
schema = load_schema("examples/invoice.schema.json") # or a dict, or a Pydantic model
for f in schema.fields(): # depth-first, containers first
print(f.field_id, f.path, f.type, f.required, f.instructions)
schema.by_path("line_items[3].amount") # instance paths are accepted
schema.get("fld_…")
schema.to_dict()
schema.schema_hash
load_schema normalizes first (local $defs inlined, anyOf[X, null] → X, type: [X, "null"]
→ X, const → one-value enum, default/examples/additionalProperties: false dropped),
then validates and raises SchemaValidationError with every SchemaIssue{pointer, code,
message, suggestion} (JSON Pointer into the normalized schema; suggestions by similarity).
Codes include unsupported_combinator (oneOf/anyOf with more than one non-null type,
allOf, not, if/then), external_ref, unresolved_ref, ref_cycle, empty_enum,
invalid_enum, unsupported_format, unsupported_keyword, max_depth_exceeded,
duplicate_key (two keys equal after snake_case, e.g. Invoice Number and invoice_number),
instructions_too_long, invalid_field_id, duplicate_field_id, no_fields, missing_items,
invalid_x_dx. Finally it assigns a fld_ id to every field without one: random by default,
deterministic with id_namespace=. Store schema.to_dict() so the ids persist; renaming a key
while keeping its x-dx.field_id keeps its identity.
The published metaschema (schemas/extraction-schema-meta-1.2.json (-1.1 stays)) lets any JSON Schema
library check a stored schema; the loader adds what JSON Schema cannot express: depth,
snake_case collisions, unique field_id, enum_descriptions keys and table placement.
From Pydantic
from typing import Literal
from pydantic import BaseModel
from extractit.core.schema import dx_field, load_schema
class LineItem(BaseModel):
description: str
amount: float | None = None
class Invoice(BaseModel):
invoice_number: str = dx_field(instructions="Starts with INV-", evidence="required")
currency: Literal["EUR", "USD"] | None = None
supplier_vat_id: str | None = dx_field(None, format="vat_id")
total: float
line_items: list[LineItem] = dx_field(default_factory=list, table={"completeness": "all_rows"})
schema = load_schema(Invoice)
Optional becomes nullable X, Literal an enum, nested models are inlined and Pydantic's
auto-generated titles are dropped, so load_schema(Invoice) from examples/invoice_model.py is
structurally equivalent to examples/invoice.schema.json. Field ids are derived from
sha256("pydantic:<module>.<qualname>:<path>"), so reloading a model keeps them. Limitations:
Decimal (a number-or-string union) and recursive models are rejected; use float and a bounded
structure.
Examples
examples/invoice.schema.json, receipt.schema.json and contract.schema.json cover every
type (string, number, integer, date, date-time, boolean, enum, multi-select, arrays of primitives
and of objects, nested objects) and are stored with their field_ids.
Synthetic documents for tests: extractit.testing.docir_factory
DocIRBuilder builds valid DocIR without a parser or a PDF. Words are laid out with a fixed
character width, so ground-truth boxes are exact. T07, T08 and T09 use it.
from extractit.core.geometry import iou
from extractit.testing.docir_factory import DocIRBuilder
b = DocIRBuilder() # char_width=0.008, line_height=0.012
b.page().block("title").line("FACTURA FAC-2026-0042", y=0.03)
b.block() # following lines share one paragraph
for i in range(36):
b.line(f"Concepto {i + 1:02d} servicio {i + 10},00 €") # y advances automatically
b.block().line("Total: 4.594,62 €", x=0.08, y=0.82)
b.table([["Item", "Amount"], ["Widget", "10,00"]], y=0.86, col_widths=0.2)
b.page(ocr=True, rotation=90).line("Scanned page") # OCR words get conf=0.95
doc = b.build() # ParsedDocument, ids deterministic
box = b.gt_bbox("4.594,62") # union of the words the text overlaps
word = next(w for w in doc.words if w.text == "4.594,62")
assert iou(box, word.bbox) == 1.0
Anchors are numbered per page (p1_l1…, p1_b1…, p1_t1); a table puts one line per non-empty
cell in its own table block and fills TableStructure with cell boxes and word indices.
ChangeSet.evidence (06 §14)
ChangeSet.evidence[] held references to the session's run_sample actions
({action_id, tool, documents}). From T44 it also carries the measurement of the change; the
earlier entries are unchanged, so a reader that only knew them keeps working:
| Entry | Shape |
|---|---|
| measured | {kind: "eval", status: "ready", run_request_id, before: evr_…, after: evr_…, compare, cost_delta, n, reused_before, snapshot_id} |
| in progress | {kind: "eval", status: "pending_approval" \| "running", run_request_id, n} |
| unmeasured | {kind: "eval", unmeasured: true, reason, run_samples: [aga_…]} (failed adds error) |
before and after are evaluations of the same snapshot and scope (validation, at most 20
documents); compare is T38's GET /v1/evaluations:compare?a=<after>&b=<before> verbatim (every
difference is after - before, 95 % intervals, Holm-adjusted tests); cost_delta is
{before_usd, after_usd, delta_usd, spent_usd} (spent_usd excludes a reused base evaluation) or
{estimated_usd} when no ledger is wired. approval.evidence is "measured" or "none".
RunRequest (rr_…): {kind: eval|optimize, status: proposed|approved|auto|running|done|rejected|expired,
est_cost_usd, docs, session_id, changeset_id?, approval, result}.