Datasets and evaluations
A dataset is a set of ground-truth items of one workflow: a document, its expected extraction (the ground truth, versioned), tags, a split and where it came from. An evaluation scores one published version of the workflow on one or more snapshots of datasets: numbered (#1, #2… per workflow), immutable once finished, with accuracy per field, per document and per tag, and expected versus obtained for every field. The contracts for other modules are in Quality contracts; the scorers and the bootstrap are those of Evaluation (extractit eval).
Creating a dataset
curl -X POST $DX/v1/datasets -H "Authorization: Bearer $KEY" \
-d '{"workflow_id": "wf_…", "name": "facturas-2026", "seed": 7}'
A workflow can have several datasets. Splits are 40 / 20 / 40 (proportions) for train,
validation and test. They are stratified by tag, deterministic per seed, and items with the
same content (sha256) or the same group_key never straddle two splits. A new item goes to the
split furthest below its target in its stratum and no existing item ever moves; only
POST /v1/datasets/{id}:resplit (with a mandatory reason) deals everything again and creates a
new snapshot. A dataset with fewer than 10 items in validation or test carries the warning
SPLIT_TOO_SMALL.
Uploading ground truth
curl -X POST "$DX/v1/datasets/ds_…/items?dry_run=true" -H "Authorization: Bearer $KEY" \
-F files=@a.pdf -F files=@a.json -F files=@b.pdf -F files=@b.json -F tags=2026,es
Pairs are matched by base name (a.pdf + a.json); the JSON is the expected extraction. It is
checked against the output contract of the workflow (the extraction schema of its extract node):
| in the JSON | gt_field_status |
|---|---|
| the key has a value | labelled |
the key is null |
absent (the document does not have this field: any answer is a hallucination) |
| the key is missing | unlabelled (nobody labelled it: it counts nowhere) |
If any file violates the contract the whole upload is refused with 422 GT_INVALID, listing every
violation (pointer, rule) and creating nothing; PDFs without JSON and JSONs without PDF come
back in unpaired. ?dry_run=true validates and creates nothing.
Items also arrive from the service (DatasetService.add_item, used by the promotion of reviewed
documents) with origin uploaded, synthetic or promoted. The item's packet is its own logical
copy (same blobs): deleting the production packet does not touch the dataset.
Editing, tags, sub-datasets, snapshots
PUT …/items/{item}/ground-truthwithIf-Match: "v<n>"creates versionn + 1; the history (GET …/history) keeps every version and a staleIf-Matchis 412.GET /v1/datasets/{id}/tagscounts items per tag, withuntagged.POST …/subsetssaves a filter (tags_all,tags_any,splits): a sub-dataset is a view.POST …/snapshotsfreezes the items with their ground-truth version, split and tags under acontent_hash(SHA-256 of the RFC 8785 form of the list); asking again for unchanged content returns the same snapshot (200). Evaluations reference snapshots, so editing a ground truth never changes a past evaluation.
The sealed test split
The optimizer, the agent and the calibration never read the items or ground truth of test
(403 TEST_SPLIT_SEALED, by API, by service and by direct repository call); people read it and each
read is logged (GET …/test-split-access). Sending X-DX-Access-Context: agent (or optimizer,
calibration) marks a call as one of those; the header can only narrow access.
Evaluations
curl -X POST $DX/v1/evaluations -H "Authorization: Bearer $KEY" -d '{
"alias": "facturas@dev", "snapshot_ids": ["dss_…", "dss_…"],
"scope": {"splits": ["validation"], "tags": ["es"]}}'
- The default scope is
train+validation;testneeds"confirm_test": true(422CONFIRM_TEST_REQUIREDwithout it) and is logged. - With several snapshots the metrics are over their union;
per_datasetbreaks them down and an item present in two datasets is flaggedduplicate_across_datasetsand counted once in the union. GET /v1/evaluations/{id}hasmetrics(T10's metrics with 95 % bootstrap intervals,per_field,per_tag),GET …/fieldsthe expected-versus-obtained rows (filter byclass,tag,path,split; keyset pages),GET …/itemsthe per-document results andGET …/tagsthe per-tag accuracy. A run that failed counts as a failed document with every labelled field omitted.GET /v1/evaluations:compare?a=&b=is the paired comparison of T10 (bootstrap of the difference, exact McNemar test with Holm's correction); 422SNAPSHOTS_DIFFERif the two evaluations were not made on the same snapshots and scope.
Workflows with Classify and Split
Create the dataset with "gt_kind": "segmented" (the workflow must have a Classify or Split node,
otherwise 422 EVAL_NOT_APPLICABLE) and upload ground truth shaped as
{"segments": [{"id", "category", "pages": [...], "data": {...}}], "unassigned_pages": [...]}. The
evaluation adds classification (accuracy, macro F1, confusion matrix with unclassified) and
partition metrics (boundary F1, first-page F1, segment IoU, exact segments, perfect packets, Pk,
WindowDiff) under metrics.classification and metrics.segments, and scores the fields of each
segment after pairing it with the ground-truth segments (a missing segment's fields are omissions,
an extra segment counts in spurious_segments). Pk and WindowDiff are unreliable when segments
average fewer than 4 pages (unreliable_short_segments).