Quality contracts (datasets, evaluations, segments)
Contracts of the quality layer (extractit.quality, T38) for the tickets that build on it:
T37 (Studio: Health), T39 (calibration), T40 (learning data), T41 (optimizer) and
T42 (learned routing and few-shot). The user guide is Datasets and evaluations.
AccessContext and the sealed test split
from extractit.quality.datasets.sealed import AccessContext, AccessKind, SealedSplitGuard
ctx = AccessContext(AccessKind.OPTIMIZER, actor="opt_01", reason="round 3")
SealedSplitGuard.assert_visible(ctx, "test") # raises SealedSplitError → 403 TEST_SPLIT_SEALED
AccessKind |
reads test? |
logged in test_split_access? |
|---|---|---|
agent, optimizer, calibration |
never (403 TEST_SPLIT_SEALED) |
— |
human, learning, final_report |
yes | yes (actor, context, reason, resource, time) |
Rules for callers:
- Every read of items or ground truth in
DatasetRepositoryandDatasetServicetakes anAccessContext(items_stmt,item,gt_version,gt_history,gt_versions_for,get_item,get_ground_truth,update_ground_truth). The guard is in the service and the repository; a direct repository call with a sealed context raises too. - A listing without a
splitfilter returns only what the context may read (notestfor the sealed kinds); asking fortestexplicitly raises. - T41, T42, T39 and the agent must create their evaluations with their own kind
(
EvaluationService.create(..., ctx=AccessContext(AccessKind.OPTIMIZER, ...), trigger="optimizer")): a scope withtestis then 403, and the default scope istrain+validation. The kind is stored inevaluation_runs.config.access, so the runner reads ground truth with it. - Over HTTP the default context is
human.X-DX-Access-Context: agent|optimizer|calibrationcan only narrow access (a runner marks its calls with it); no header can widen it.X-DX-Access-Reasonis the reason logged for a person's read oftest. learning(T40/T42) andfinal_reportare internal contexts: they are not reachable over HTTP.- Without a
group_key, near-identical templates can end up in different splits and inflate the figures oftest; setgroup_key(or the same template tag) when the corpus has templates.
DatasetService.add_item (T37's promotion)
DatasetService(session, tenant_id, project_id, actor=..., services=ingest_services).add_item(
dataset_id, # first argument: the dataset to add to
packet, # uuid.UUID of an existing packet (logical copy) or a list of `Incoming` files
ground_truth, # the extraction as a JSON object (plain) or {segments, unassigned_pages}
origin, # "uploaded" | "synthetic" | "promoted"
tags=(),
*, group_key=None, name=None, provenance=None, require_complete=None,
) -> DatasetItem
- A UUID
packetcreates a new packet with the same blobs by sha256 (no bytes copied): deleting the production packet never alters the item or its ground truth. ground_truthis validated against the output contract of the workflow's latest published version (all violations,GtInvalidError);gt_field_statusis derived: key with a value →labelled,null→absent, missing key →unlabelled.origin="promoted"requires every field to have a decision (nounlabelled):PromotionIncompleteError→ 409PROMOTION_INCOMPLETEwith the field paths (409 since T37: the document is not ready; the promotion route of T37,quality/datasets/promote.py, reads the effective result of T32 and callsadd_item(origin="promoted")).- The signature takes
dataset_idfirst (the ficha'sadd_item(packet, ground_truth, origin, tags)does not say which dataset). Changing this signature obliges T37 to adaptquality/datasets/promote.py.
Evaluations
POST /v1/evaluations→ 202 withnumber(#1, #2… per workflow, fromevaluation_counters).scope.splitsdefaults totrain+validation;testneedsconfirm_test=true(422CONFIRM_TEST_REQUIRED) and is logged.- Each item runs as a run of the interpreter with trigger
{"kind": "eval", "purpose": "eval"}(InterpreterItemExecutor(purpose=...): T41 passesoptimizer, T42learning, so T40 leaves those runs out of its learning data). The durable run kind isevaluation(extractit.quality.handlers); the coupling with T26 is confined toquality/evaluations/runner.py. - Completed and failed evaluations are immutable (trigger; see below). The metrics are computed
from the stored rows, so re-scoring them reproduces every number
(
results.compute_metrics(EvaluationRepository.stored_items(id), n_boot=…, seed=…)).
Columns of evaluation_field_results
| column | meaning |
|---|---|
evaluation_id, item_result_id, dataset_id, item_id |
what the row belongs to |
split |
train, validation or test (the sealed contexts never read test rows) |
segment_id |
ground-truth segment id for segmented items; '' (null in the API) otherwise |
path |
field path (total, supplier.name); table cells are in the item's record |
expected, obtained |
JSON values (expected null when the field is absent; obtained null for no answer) |
gt_status |
labelled or absent (unlabelled fields have no row: they are in no denominator) |
class |
correct, omission, hallucination, wrong_value, format_only (right after normalization, written differently) |
outcome |
T10's outcome: tp, fp_value, fn_omission, fp_hallucination, tn |
confidence, scores_kind |
heuristic confidence of the system (heuristic, calibrated after T39, none) |
citation_status |
verdict of the primary evidence (verified, partial…) or null |
anls, ftype |
similarity of strings and the field type |
evaluation_item_results has one row per item: status (ok / failed: a failed run counts as a
failure with every labelled field omitted), perfect, accuracy, duplicate_across_datasets,
record (T10's record JSON: tables, citations, operation) and segmentation (partition scores and
the page classification of segmented items).
Immutability
evaluation_runs has a BEFORE UPDATE OR DELETE trigger: once completed or failed nothing
changes and a direct DELETE is refused; while it runs its identity (number, version, snapshots,
scope, config_hash) cannot change. The result tables are append-only for dx_app and refuse
UPDATE always. A DELETE only passes in a cascade (deleting the workflow, project or tenant) or under
SET LOCAL dx.retention_purge = 'on' (immutability.retention_purge(session), reserved to T50).
dataset_item_gt_versions, dataset_snapshots and test_split_access are append-only the same way.
paired_gate (T41 and T42)
from extractit.quality.acceptance import paired_gate
result = paired_gate(
eval_base, # list[DocRecord] of the base evaluation (same snapshot)
eval_candidate, # list[DocRecord] of the candidate
metric="field_accuracy", # a T10 field metric or "perfect_documents"
protected_slices=["es", "scan"], # tags
tolerance=0.02, # absolute points a protected slice may lose
)
result.accepted # False when there is any reason
result.reasons # ["slice_regression", "gain_not_significant"]
result.lower_bound # lower end of the paired 95 % interval of the improvement
result.regressions # [SliceRegression(slice, base, candidate, delta, documents)]
- The improvement is
candidate − base(negated forhallucination_rate, where lower is better), with the paired hierarchical bootstrap of T10 (documents, then fields) and 2 000 replicas.gain_not_significant: the lower bound is not above 0.slice_regression: a protected tag worsened by more thantoleranceon the documents that carry it. - Records come from
EvaluationService.records(evaluation_id)(one T10DocRecordper item, tagged with the item's tags plussplit:<name>); only documents present in both count. - The signature is a contract: change it only if T41 and T42 agree.
- T42 uses it twice (Learning): the few-shot gate reads
accepted/lower_bound(improvement > 0), and the routing verdict readslower_boundas a non-inferiority bound (≥ −1.0 point) together with the cost saving.
Segment contract
Evaluation of workflows with Classify/Split uses the segment contract, not the real Split node
(T28). If T28 changes it, adapt quality/segments/contract.py and the fake executor.
Segmented ground truth (dataset.gt_kind = "segmented"):
{"segments": [{"id": "s1", "category": "invoice", "pages": [1, 3],
"partition_value": "ACC-1", "data": {"invoice_number": "INV-1"}}],
"unassigned_pages": [5]}
Pages are 1-based packet pages and may be non-consecutive (A1, B1, A2, B2). Validation reports
every violation: unknown_category, duplicate_id, duplicate_page, page_in_two_segments,
page_out_of_range, shape errors and the schema violations of each segment's data
(pointer /segments/<i>/data/...). gt_field_status of a segmented ground truth is a map per
segment id.
What a run reports for its segments is DocumentSegment (06 §8): rule_id, partition_value,
pages: [{file_id, file_page, packet_page}] and status (assigned / unassigned);
outputs.segments[node_id] is the list and outputs.extractions[extract_node].scopes[segment_id]
holds the extraction of each segment (the scope key is the segment id).
Scores (quality/segments/scorers.py): page classification (accuracy, macro F1 over the labels in
the reference or the prediction, confusion matrix with unclassified), boundary F1 and first-page F1
(tolerance 0, boundary = page i whose segment differs from page i + 1), IoU of segments, exact
segment rate, perfect packets, Pk and WindowDiff with k = round(N / (2·#segments_ref)) and the
unreliable_short_segments flag below 4 pages per segment. Segments are paired by the Hungarian
assignment over 1 − IoU of page sets with threshold 0.5: the fields of an unpaired ground-truth
segment are omissions; unpaired predicted segments go to spurious_segments.
Errors added to the catalog
TEST_SPLIT_SEALED (403), GT_INVALID, CONFIRM_TEST_REQUIRED, SNAPSHOTS_DIFFER,
EVAL_NOT_APPLICABLE (422), PROMOTION_INCOMPLETE, EVALUATION_NOT_FINISHED,
DATASET_NAME_TAKEN (409).