Skip to content

Core contracts

extractit.core holds the types every module shares, so that parsing, extraction, verification, rules and evaluation can be built in parallel against fixtures. All models are Pydantic v2, frozen and strict about unknown keys (extra="forbid"). Each contract is published as a versioned JSON Schema in schemas/.

Contract Model Version constant Snapshot
Parsed document (DocIR) core.docir.ParsedDocument DOCIR_VERSION schemas/docir-1.2.json (docir-1.0.json and docir-1.1.json stay for older readers)
Anchored text core.anchored.AnchoredMarkdown ANCHORED_MARKDOWN_VERSION schemas/anchored-markdown-1.0.json
Extraction output core.output.ExtractionOutput OUTPUT_SCHEMA_VERSION schemas/output-1.2.json (output-1.0.json and output-1.1.json stay for older readers)
Extraction-schema dialect core/schema/metaschema.json EXTRACTION_SCHEMA_DIALECT_VERSION schemas/extraction-schema-meta-1.2.json (-1.1 stays)

Versioning policy

  • Versions are MAJOR.MINOR and live in extractit/core/version.py.
  • An additive change (new optional field, new enum value a reader may ignore) bumps MINOR. An incompatible change (removed or renamed field, narrower type) bumps MAJOR and needs a migration note in CHANGELOG.md.
  • Until phase F1 closes, every contract stays at 1.x: only additive changes.
  • A version that is not in any release (no tag) absorbs further additive changes: regenerate its snapshot in place, without bumping another MINOR. Once it is published, additive = a new MINOR.
  • When you bump, also add the new value to the matching *Version literal so that documents written with an older minor keep loading, run the export and commit the new snapshot. Older snapshots stay in schemas/.
uv run python -m extractit.core.schemas export   # write schemas/<name>-<version>.json
uv run python -m extractit.core.schemas check    # exit 1 if a model changed without a bump
make contracts                                       # runs scripts/contracts.d/*.sh (CI)

check exits 1 when the snapshot for the current version is missing or differs from what the models generate. If only descriptions (docstrings) changed, it says so: re-export, no bump.

Identifiers and geometry

  • Ids are TypeIDs: <prefix>_<26 base32 chars> over a UUIDv7 (pdoc_, fld_, ev_, vr_, mc_, rule_, ext_, el_, tbl_…). new_id(prefix) is time-sortable; deterministic_id(prefix, key) hashes key (sha256, first 16 bytes, UUIDv7 version/variant bits) and always returns the same valid id: the pipeline uses it so that two runs with the fake model produce identical output. id_to_uuid/id_from_uuid convert for uuid columns.
  • Coordinates are normalized to the page as displayed (after /Rotate and the CropBox), origin top-left, 0 ≤ x0 < x1 ≤ 1, 0 ≤ y0 < y1 ≤ 1 (BBox). JSON output rounds boxes outwards to 4 decimals. to_pdf_space(bbox, page) / from_pdf_space(rect, page) convert to PDF user space (points, origin bottom-left) applying the inverse rotation first, then the CropBox. iou and union_bbox are shared by the verifier and evaluation.
  • Canonical JSON: serialize(obj) (sorted keys, compact, UTF-8) gives identical bytes for identical objects; canonical_hash is its sha256.

DocIR 1.2 (ParsedDocument)

ParsedDocument{docir_version, id: pdoc_…, sha256, parse_cache_key, profile{tier, profile_hash, options},
               pages[], words[], lines[], blocks[], tables[], logical_tables[], outline[],
               quality{parse_confidence, layout_confidence, pages_flagged, warnings},
               provenance{engines[], model_calls[], started_at, finished_at, cache_hit}}
Page{number, width_pt, height_pt, rotation_applied, deskew_angle_deg, crop_box?, render?,
     text_source, route, quality{text_layer_coverage, ocr_mean_conf, parse_confidence, …, complexity}}
Word{idx, page, text, bbox, polygon?, conf, source, line_idx, block_idx, font?}
Line{idx, id: el_…, anchor: p{n}_l{m}, page, bbox, text, word_range[start, end), block_idx}
Block{id: el_…, anchor: p{n}_b{m}, type, page, bbox, order, level, parent_id, text, markdown,
      line_range, confidences{text, layout}, source{engine, model_version, route}}
TableStructure{id: tbl_…, anchor: p{n}_t{k}, page, block_idx, n_rows, n_cols, header_rows,
               cells[{row, col, row_span, col_span, text, bbox, is_header, word_idxs, conf,
                      word_range?}],
               column_x, method, confidence, continues?{from, to}, extra{}}
LogicalTable{id: ltbl_…, segments[{table_id, page, packet_page?, row_offset, header_repeated,
             data_rows[]}], header, header_levels, n_rows, stitch_confidence, stitch_evidence[],
             stitch_checks[{name, passed, detail}], split_rows[{row, parts[RowPart]}]}
RowPart{table_id, page, packet_page?, physical_row};  LogicalTable.row_ref(i) -> RowRef{row, parts}

DocIR 1.1 (T19, additive). TableCell.word_range is set only when the cell's words (word_idxs, the word_ids of the T19 ticket) are contiguous; TableStructure.continues links a table to the previous (from) and next (to) segment of its logical table and extra carries open flags (nested, orphan_words, refined, structure_score); LogicalTable gains header_levels, stitch_checks, split_rows, per-segment data_rows and packet_page, and stitch_evidence gains geometry, numbering, column_types and override. Cell anchors p{n}_t{k}_r{j}_c{c} (CELL_ANCHOR_PATTERN) appear as data-c in table HTML and resolve with doc.resolve_anchor(anchor) → ResolvedAnchor{anchor, page, word_ids, bbox} (row anchors give the union of the row's cells; an unknown anchor raises AnchorNotFound). They are not AnchorRef entries: the anchored-text contract is unchanged. Documents written as 1.0 still load (tests/fixtures/contracts/compat/docir-1.0-2p-120w.json). See Tables.

DocIR 1.2 (T20, additive) adds the VLM alignment of route vlm_page (see VLM per page).

DocIR 1.2 also absorbs T21 (additive; 1.2 is not in any release yet, so its snapshot is regenerated in place, no 1.3): the page router and path D (see Page router): TableCell.text_source (words | vlm_c | vlm_d | vote), TableCell.alternatives ([{source, text}], sources words | vlm_c | vlm_d | vlm_d2 | vlm_x) and TableCell.flags (conflict | no_majority | numeric_conflict) on cells reconciled by path D; PageQuality.cross_engine_agreement and DocumentQuality.cross_engine_agreement; Provenance.routing (one PageRouting{page, path, route, engine, reason, expected_cost_usd, complexity, signals, route_history[]} per page) and Provenance.fallbacks (PageFallback{page, from, to, cause}). Every new field is omitted from the JSON when unset, so a document parsed without the router serializes as before. The router's own contracts live in schemas/router-{policy,routing,table-reading,parse-problem}-1.0.json (scripts/contracts.d/router.sh).

ParseRoute is text_layer | ocr | vlm_page | vlm_agentic | table_specialist | external | none. Pages are 1-based; words, lines and blocks are addressed by their position. Construction fails — naming the culprit, e.g. words[7].line_idx — unless:

  • every word belongs to exactly one line, line word ranges tile words in order, and words agree with their line on page and block;
  • idx fields are dense and equal to the position; page numbers are 1..N;
  • anchors match their format, name the right page and are unique; element ids are unique;
  • Block.order is a permutation of 0..N-1; line/block/table/parent references exist (also continues, logical-table segments, their data_rows and split-row parts);
  • every box is a valid BBox in [0, 1].

Row anchors p{n}_t{k}_r{j} are only valid in anchored text and evidence (rows are not elements).

DocIR 1.2 (T24, additive). Page gains file_id, file_page and packet_page: in a packet (several files seen as one document) the pages are numbered globally, anchors become p{packet_page}_l{n} and each page records the file and the page inside it. They are null outside a packet. See Field sources.

DocIR 1.2 (T20, additive). The VLM alignment of route vlm_page: Line.alignment (provenance ladder, score, flags, polygon, character spans to words and alternative readings), Block.alignment, the alignment metrics of PageQuality, Provenance.cost and ParsedDocument.words_for_span. Every new field is omitted from the JSON when unset, so a document without VLM alignment serializes as in 1.1. FieldEvidence.anchor also accepts a cell anchor (T22).

Anchored text 1.0 (AnchoredMarkdown)

The only bridge between the renderer (T04, sole producer) and the extraction harness (T06): T06 and T08/T10 never need the parser.

AnchoredMarkdown{anchored_markdown_version, renderer_version, granularity: line|block|both,
                 doc_sha256, text,
                 anchors[{anchor, kind: line|block|table|row, page, line?, block?, char_start, char_end}],
                 page_map[{page, char_start, char_end}]}

text[char_start:char_end] is the anchored content. Anchors are unique and well formed, their kind and page match the anchor name, spans are non-empty, inside text and inside their page; page_map lists pages in increasing order and tiles text with no gap or overlap. Helpers: am.get(anchor), anchor in am, am.anchor_text(anchor), am.page_text(page).

Results and output 1.2

  • FieldCandidate{path, field_id, status_hint: extracted|not_found|abstained, value, raw_text, anchors, quote, model_call_id} — what the harness (T06) proposes; T07 verifies it.
  • FieldEvidence{id: ev_…, kind, role, page, anchor, quote, char_span, word_range, words[{text, bbox, word_idx}], regions[{page, bbox, polygon}], table_ref, verification, match{method, score, normalized_value}, citation_corrected, produced_by}.
  • ConfidenceScore{score, kind: calibrated_probability|heuristic, band, signals, calibrator?, threshold_policy_id?, reason?}.
  • ValidationResult{id: vr_…, rule_id, rule_version, status: pass|fail|inconclusive|skipped, severity, blocking, detail, compared_values, evidence_ids, inconclusive_reason, model_call_id}; inconclusive_reason is only allowed when status is inconclusive.
  • Usage{model, model_snapshot, billing: metered|subscription|agent|none, input_tokens, output_tokens, cost_usd, duration_ms, steps[]}.
  • ExtractedField{path, field_id, value, raw_text, status, confidence, evidence, alternatives, review_state}: every field is nullable; not_found has value = null, extracted does not.

ExtractionOutput{schema_version, extraction{mode: llm|agent, tier, model, agent}, document{sha256, pages, routes, pdoc_id, filename}, data, fields, document_confidence, operator_confidence{parse, extract, validate}, requires_review, validations, usage, reproducibility{schema_hash, prompt_hash, model_snapshot, engine_versions, parse_cache_key, docir_version, tier, seeds, deterministic}, warnings, strategy, coverage} is what extractit FILE prints. Field paths and evidence ids are unique and validations only cite evidence present in fields. The example in tests/fixtures/contracts/valid/output-4-fields.json shows every part.

Output 1.1 (T23, additive). strategy is single_pass (default: one request over the whole text) or long_document (location, windows and consolidation, see Long documents); coverage{pages_covered, pages_total, pages_read, failed_windows} is set only by the long-document strategy and pages_covered < pages_total means a window failed. Outputs written as 1.0 still load.

Output 1.2 (T22 and T24, additive). table_stats{path: {rows_physical, rows_extracted, rows_excluded, rows_merged, completeness, missing_rows, mode, tables, stitch_suspect, anomalous_rows}} is set when an x-dx.table field was extracted (see Complete tables); null otherwise. table_cell evidence may carry a cell anchor (p3_t1_r17_c4) in anchor and one region per physical row part (a row cut by a page break cites two pages). The rules contract 1.0 gains row_count with against: table_stats and table (additive). FieldEvidence also gains lookup_ref{table_id, table_version, row_key, matched_on, match_score} (kind='lookup_row'), derivation{expression, inputs} (kind='derived'), metadata_path (kind='metadata') and file_id, file_page, packet_page (evidence on a packet); ConfidenceSignals gains source_confidence (the match score times the weakest input, or the weakest input of a derived field, which replaces the grounding base); a FieldCandidate may cite metadata.<path> and notes.l<n> anchors. Outputs written as 1.0 and 1.1 still load. See Field sources. output-1.1.json stays as it was published, for older readers.

Output 1.2 (T39). ConfidenceScore.reason (optional): calibrator_stale when a calibrator exists for the field but its model snapshot or prompt hash differ from the run's, so the score stays heuristic. A calibrated_probability score carries calibrator{id, version, scope{field_type, model_snapshot}, ece, n_samples, fitted_at}; see Calibrated confidence. Outputs written as 1.0 or 1.1 still load.

Ports: ExtractionView and ValidatorPort

The pipeline (T08) and the rules engine (T09) only meet here. ExtractionView(fields, data=None) (or ExtractionView.from_output(out)) is read-only: get/value/confidence/evidence_ids by field_id or path, select("line_items[*].amount") and by_field_id(id) for every row, row_count("line_items"), and data (rebuilt from the fields with data_from_fields if not given). A rules engine implements:

class ValidatorPort(Protocol[RuleT]):
    def evaluate(
        self, rules: Sequence[RuleT], view: ExtractionView, /
    ) -> list[ValidationResult]: ...

Extraction-schema dialect 1.2

1.2 (T24, additive). x-dx.source is a string (the legacy values) or an object {kind, …}: document, lookup, derived, metadata (and kinds registered by other packages). The legacy strings with lookup, expression or metadata next to them normalize to the object (FieldDef.source_spec; FieldDef.source keeps a flat label). See Field sources.

Restricted JSON Schema 2020-12 plus x-dx extensions. Allowed keywords on a field: type, properties, required, items, enum, format, description, title, minimum, maximum, x-dx. Types: string (formats date, date-time, email, uri, iban, bic, vat_id, phone), number, integer, boolean, string enum, array of primitives or objects (an array of enum strings is a multi_select), object. Nesting up to max_depth (4 by default; x-dx.max_depth at the root or load_schema(..., max_depth=)).

x-dx key Meaning
field_id Persistent identity (fld_ TypeID); survives renames. Assigned if absent.
instructions Natural-language guidance, ≤ 2 000 characters.
source document (default), lookup, lookup_if_missing, metadata, derived (executed from T24).
evidence required (no verifiable evidence ⇒ abstained), preferred (default), none.
normalize date_order, decimal_separator, thousands_separator, unit, case, strip, percent (points | fraction, since 1.1), enum_threshold (0–100, since 1.1).
table Arrays of objects only: completeness (all_rows/best_effort), row_key, hint.
lookup, expression Reference table / CEL expression for lookup/derived sources (T24).
enum_descriptions Description per enum option.
pii none, personal, special_category.

Schema-level x-dx: schema_version, document_type, locale_hint, max_depth.

Every field is nullable: required means expected — its absence is a finding, never a reason to invent a value.

Loading

from extractit.core.schema import load_schema

schema = load_schema("examples/invoice.schema.json")  # or a dict, or a Pydantic model
for f in schema.fields():  # depth-first, containers first
    print(f.field_id, f.path, f.type, f.required, f.instructions)
schema.by_path("line_items[3].amount")  # instance paths are accepted
schema.get("fld_…")
schema.to_dict()
schema.schema_hash

load_schema normalizes first (local $defs inlined, anyOf[X, null] → X, type: [X, "null"] → X, const → one-value enum, default/examples/additionalProperties: false dropped), then validates and raises SchemaValidationError with every SchemaIssue{pointer, code, message, suggestion} (JSON Pointer into the normalized schema; suggestions by similarity). Codes include unsupported_combinator (oneOf/anyOf with more than one non-null type, allOf, not, if/then), external_ref, unresolved_ref, ref_cycle, empty_enum, invalid_enum, unsupported_format, unsupported_keyword, max_depth_exceeded, duplicate_key (two keys equal after snake_case, e.g. Invoice Number and invoice_number), instructions_too_long, invalid_field_id, duplicate_field_id, no_fields, missing_items, invalid_x_dx. Finally it assigns a fld_ id to every field without one: random by default, deterministic with id_namespace=. Store schema.to_dict() so the ids persist; renaming a key while keeping its x-dx.field_id keeps its identity.

The published metaschema (schemas/extraction-schema-meta-1.2.json (-1.1 stays)) lets any JSON Schema library check a stored schema; the loader adds what JSON Schema cannot express: depth, snake_case collisions, unique field_id, enum_descriptions keys and table placement.

From Pydantic

from typing import Literal
from pydantic import BaseModel
from extractit.core.schema import dx_field, load_schema


class LineItem(BaseModel):
    description: str
    amount: float | None = None


class Invoice(BaseModel):
    invoice_number: str = dx_field(instructions="Starts with INV-", evidence="required")
    currency: Literal["EUR", "USD"] | None = None
    supplier_vat_id: str | None = dx_field(None, format="vat_id")
    total: float
    line_items: list[LineItem] = dx_field(default_factory=list, table={"completeness": "all_rows"})


schema = load_schema(Invoice)

Optional becomes nullable X, Literal an enum, nested models are inlined and Pydantic's auto-generated titles are dropped, so load_schema(Invoice) from examples/invoice_model.py is structurally equivalent to examples/invoice.schema.json. Field ids are derived from sha256("pydantic:<module>.<qualname>:<path>"), so reloading a model keeps them. Limitations: Decimal (a number-or-string union) and recursive models are rejected; use float and a bounded structure.

Examples

examples/invoice.schema.json, receipt.schema.json and contract.schema.json cover every type (string, number, integer, date, date-time, boolean, enum, multi-select, arrays of primitives and of objects, nested objects) and are stored with their field_ids.

Synthetic documents for tests: extractit.testing.docir_factory

DocIRBuilder builds valid DocIR without a parser or a PDF. Words are laid out with a fixed character width, so ground-truth boxes are exact. T07, T08 and T09 use it.

from extractit.core.geometry import iou
from extractit.testing.docir_factory import DocIRBuilder

b = DocIRBuilder()  # char_width=0.008, line_height=0.012
b.page().block("title").line("FACTURA FAC-2026-0042", y=0.03)
b.block()  # following lines share one paragraph
for i in range(36):
    b.line(f"Concepto {i + 1:02d} servicio {i + 10},00 €")  # y advances automatically
b.block().line("Total: 4.594,62 €", x=0.08, y=0.82)
b.table([["Item", "Amount"], ["Widget", "10,00"]], y=0.86, col_widths=0.2)
b.page(ocr=True, rotation=90).line("Scanned page")  # OCR words get conf=0.95

doc = b.build()  # ParsedDocument, ids deterministic
box = b.gt_bbox("4.594,62")  # union of the words the text overlaps
word = next(w for w in doc.words if w.text == "4.594,62")
assert iou(box, word.bbox) == 1.0

Anchors are numbered per page (p1_l1…, p1_b1…, p1_t1); a table puts one line per non-empty cell in its own table block and fills TableStructure with cell boxes and word indices.

ChangeSet.evidence (06 §14)

ChangeSet.evidence[] held references to the session's run_sample actions ({action_id, tool, documents}). From T44 it also carries the measurement of the change; the earlier entries are unchanged, so a reader that only knew them keeps working:

Entry Shape
measured {kind: "eval", status: "ready", run_request_id, before: evr_…, after: evr_…, compare, cost_delta, n, reused_before, snapshot_id}
in progress {kind: "eval", status: "pending_approval" \| "running", run_request_id, n}
unmeasured {kind: "eval", unmeasured: true, reason, run_samples: [aga_…]} (failed adds error)

before and after are evaluations of the same snapshot and scope (validation, at most 20 documents); compare is T38's GET /v1/evaluations:compare?a=<after>&b=<before> verbatim (every difference is after - before, 95 % intervals, Holm-adjusted tests); cost_delta is {before_usd, after_usd, delta_usd, spent_usd} (spent_usd excludes a reused base evaluation) or {estimated_usd} when no ledger is wired. approval.evidence is "measured" or "none". RunRequest (rr_…): {kind: eval|optimize, status: proposed|approved|auto|running|done|rejected|expired, est_cost_usd, docs, session_id, changeset_id?, approval, result}.