Skip to content

Quality contracts (datasets, evaluations, segments)

Contracts of the quality layer (extractit.quality, T38) for the tickets that build on it: T37 (Studio: Health), T39 (calibration), T40 (learning data), T41 (optimizer) and T42 (learned routing and few-shot). The user guide is Datasets and evaluations.

AccessContext and the sealed test split

from extractit.quality.datasets.sealed import AccessContext, AccessKind, SealedSplitGuard

ctx = AccessContext(AccessKind.OPTIMIZER, actor="opt_01", reason="round 3")
SealedSplitGuard.assert_visible(ctx, "test")  # raises SealedSplitError → 403 TEST_SPLIT_SEALED
AccessKind reads test? logged in test_split_access?
agent, optimizer, calibration never (403 TEST_SPLIT_SEALED) —
human, learning, final_report yes yes (actor, context, reason, resource, time)

Rules for callers:

  • Every read of items or ground truth in DatasetRepository and DatasetService takes an AccessContext (items_stmt, item, gt_version, gt_history, gt_versions_for, get_item, get_ground_truth, update_ground_truth). The guard is in the service and the repository; a direct repository call with a sealed context raises too.
  • A listing without a split filter returns only what the context may read (no test for the sealed kinds); asking for test explicitly raises.
  • T41, T42, T39 and the agent must create their evaluations with their own kind (EvaluationService.create(..., ctx=AccessContext(AccessKind.OPTIMIZER, ...), trigger="optimizer")): a scope with test is then 403, and the default scope is train + validation. The kind is stored in evaluation_runs.config.access, so the runner reads ground truth with it.
  • Over HTTP the default context is human. X-DX-Access-Context: agent|optimizer|calibration can only narrow access (a runner marks its calls with it); no header can widen it. X-DX-Access-Reason is the reason logged for a person's read of test.
  • learning (T40/T42) and final_report are internal contexts: they are not reachable over HTTP.
  • Without a group_key, near-identical templates can end up in different splits and inflate the figures of test; set group_key (or the same template tag) when the corpus has templates.

DatasetService.add_item (T37's promotion)

DatasetService(session, tenant_id, project_id, actor=..., services=ingest_services).add_item(
    dataset_id,            # first argument: the dataset to add to
    packet,                # uuid.UUID of an existing packet (logical copy) or a list of `Incoming` files
    ground_truth,          # the extraction as a JSON object (plain) or {segments, unassigned_pages}
    origin,                # "uploaded" | "synthetic" | "promoted"
    tags=(),
    *, group_key=None, name=None, provenance=None, require_complete=None,
) -> DatasetItem
  • A UUID packet creates a new packet with the same blobs by sha256 (no bytes copied): deleting the production packet never alters the item or its ground truth.
  • ground_truth is validated against the output contract of the workflow's latest published version (all violations, GtInvalidError); gt_field_status is derived: key with a value → labelled, null → absent, missing key → unlabelled.
  • origin="promoted" requires every field to have a decision (no unlabelled): PromotionIncompleteError → 409 PROMOTION_INCOMPLETE with the field paths (409 since T37: the document is not ready; the promotion route of T37, quality/datasets/promote.py, reads the effective result of T32 and calls add_item(origin="promoted")).
  • The signature takes dataset_id first (the ficha's add_item(packet, ground_truth, origin, tags) does not say which dataset). Changing this signature obliges T37 to adapt quality/datasets/promote.py.

Evaluations

  • POST /v1/evaluations → 202 with number (#1, #2… per workflow, from evaluation_counters). scope.splits defaults to train + validation; test needs confirm_test=true (422 CONFIRM_TEST_REQUIRED) and is logged.
  • Each item runs as a run of the interpreter with trigger {"kind": "eval", "purpose": "eval"} (InterpreterItemExecutor(purpose=...): T41 passes optimizer, T42 learning, so T40 leaves those runs out of its learning data). The durable run kind is evaluation (extractit.quality.handlers); the coupling with T26 is confined to quality/evaluations/runner.py.
  • Completed and failed evaluations are immutable (trigger; see below). The metrics are computed from the stored rows, so re-scoring them reproduces every number (results.compute_metrics(EvaluationRepository.stored_items(id), n_boot=…, seed=…)).

Columns of evaluation_field_results

column meaning
evaluation_id, item_result_id, dataset_id, item_id what the row belongs to
split train, validation or test (the sealed contexts never read test rows)
segment_id ground-truth segment id for segmented items; '' (null in the API) otherwise
path field path (total, supplier.name); table cells are in the item's record
expected, obtained JSON values (expected null when the field is absent; obtained null for no answer)
gt_status labelled or absent (unlabelled fields have no row: they are in no denominator)
class correct, omission, hallucination, wrong_value, format_only (right after normalization, written differently)
outcome T10's outcome: tp, fp_value, fn_omission, fp_hallucination, tn
confidence, scores_kind heuristic confidence of the system (heuristic, calibrated after T39, none)
citation_status verdict of the primary evidence (verified, partial…) or null
anls, ftype similarity of strings and the field type

evaluation_item_results has one row per item: status (ok / failed: a failed run counts as a failure with every labelled field omitted), perfect, accuracy, duplicate_across_datasets, record (T10's record JSON: tables, citations, operation) and segmentation (partition scores and the page classification of segmented items).

Immutability

evaluation_runs has a BEFORE UPDATE OR DELETE trigger: once completed or failed nothing changes and a direct DELETE is refused; while it runs its identity (number, version, snapshots, scope, config_hash) cannot change. The result tables are append-only for dx_app and refuse UPDATE always. A DELETE only passes in a cascade (deleting the workflow, project or tenant) or under SET LOCAL dx.retention_purge = 'on' (immutability.retention_purge(session), reserved to T50). dataset_item_gt_versions, dataset_snapshots and test_split_access are append-only the same way.

paired_gate (T41 and T42)

from extractit.quality.acceptance import paired_gate

result = paired_gate(
    eval_base,  # list[DocRecord] of the base evaluation (same snapshot)
    eval_candidate,  # list[DocRecord] of the candidate
    metric="field_accuracy",  # a T10 field metric or "perfect_documents"
    protected_slices=["es", "scan"],  # tags
    tolerance=0.02,  # absolute points a protected slice may lose
)
result.accepted  # False when there is any reason
result.reasons  # ["slice_regression", "gain_not_significant"]
result.lower_bound  # lower end of the paired 95 % interval of the improvement
result.regressions  # [SliceRegression(slice, base, candidate, delta, documents)]
  • The improvement is candidate − base (negated for hallucination_rate, where lower is better), with the paired hierarchical bootstrap of T10 (documents, then fields) and 2 000 replicas. gain_not_significant: the lower bound is not above 0. slice_regression: a protected tag worsened by more than tolerance on the documents that carry it.
  • Records come from EvaluationService.records(evaluation_id) (one T10 DocRecord per item, tagged with the item's tags plus split:<name>); only documents present in both count.
  • The signature is a contract: change it only if T41 and T42 agree.
  • T42 uses it twice (Learning): the few-shot gate reads accepted/lower_bound (improvement > 0), and the routing verdict reads lower_bound as a non-inferiority bound (≥ −1.0 point) together with the cost saving.

Segment contract

Evaluation of workflows with Classify/Split uses the segment contract, not the real Split node (T28). If T28 changes it, adapt quality/segments/contract.py and the fake executor.

Segmented ground truth (dataset.gt_kind = "segmented"):

{"segments": [{"id": "s1", "category": "invoice", "pages": [1, 3],
               "partition_value": "ACC-1", "data": {"invoice_number": "INV-1"}}],
 "unassigned_pages": [5]}

Pages are 1-based packet pages and may be non-consecutive (A1, B1, A2, B2). Validation reports every violation: unknown_category, duplicate_id, duplicate_page, page_in_two_segments, page_out_of_range, shape errors and the schema violations of each segment's data (pointer /segments/<i>/data/...). gt_field_status of a segmented ground truth is a map per segment id.

What a run reports for its segments is DocumentSegment (06 §8): rule_id, partition_value, pages: [{file_id, file_page, packet_page}] and status (assigned / unassigned); outputs.segments[node_id] is the list and outputs.extractions[extract_node].scopes[segment_id] holds the extraction of each segment (the scope key is the segment id).

Scores (quality/segments/scorers.py): page classification (accuracy, macro F1 over the labels in the reference or the prediction, confusion matrix with unclassified), boundary F1 and first-page F1 (tolerance 0, boundary = page i whose segment differs from page i + 1), IoU of segments, exact segment rate, perfect packets, Pk and WindowDiff with k = round(N / (2·#segments_ref)) and the unreliable_short_segments flag below 4 pages per segment. Segments are paired by the Hungarian assignment over 1 − IoU of page sets with threshold 0.5: the fields of an unpaired ground-truth segment are omissions; unpaired predicted segments go to spurious_segments.

Errors added to the catalog

TEST_SPLIT_SEALED (403), GT_INVALID, CONFIRM_TEST_REQUIRED, SNAPSHOTS_DIFFER, EVAL_NOT_APPLICABLE (422), PROMOTION_INCOMPLETE, EVALUATION_NOT_FINISHED, DATASET_NAME_TAKEN (409).