Skip to content

Datasets and evaluations

A dataset is a set of ground-truth items of one workflow: a document, its expected extraction (the ground truth, versioned), tags, a split and where it came from. An evaluation scores one published version of the workflow on one or more snapshots of datasets: numbered (#1, #2… per workflow), immutable once finished, with accuracy per field, per document and per tag, and expected versus obtained for every field. The contracts for other modules are in Quality contracts; the scorers and the bootstrap are those of Evaluation (extractit eval).

Creating a dataset

curl -X POST $DX/v1/datasets -H "Authorization: Bearer $KEY" \
     -d '{"workflow_id": "wf_…", "name": "facturas-2026", "seed": 7}'

A workflow can have several datasets. Splits are 40 / 20 / 40 (proportions) for train, validation and test. They are stratified by tag, deterministic per seed, and items with the same content (sha256) or the same group_key never straddle two splits. A new item goes to the split furthest below its target in its stratum and no existing item ever moves; only POST /v1/datasets/{id}:resplit (with a mandatory reason) deals everything again and creates a new snapshot. A dataset with fewer than 10 items in validation or test carries the warning SPLIT_TOO_SMALL.

Uploading ground truth

curl -X POST "$DX/v1/datasets/ds_…/items?dry_run=true" -H "Authorization: Bearer $KEY" \
     -F files=@a.pdf -F files=@a.json -F files=@b.pdf -F files=@b.json -F tags=2026,es

Pairs are matched by base name (a.pdf + a.json); the JSON is the expected extraction. It is checked against the output contract of the workflow (the extraction schema of its extract node):

in the JSON gt_field_status
the key has a value labelled
the key is null absent (the document does not have this field: any answer is a hallucination)
the key is missing unlabelled (nobody labelled it: it counts nowhere)

If any file violates the contract the whole upload is refused with 422 GT_INVALID, listing every violation (pointer, rule) and creating nothing; PDFs without JSON and JSONs without PDF come back in unpaired. ?dry_run=true validates and creates nothing.

Items also arrive from the service (DatasetService.add_item, used by the promotion of reviewed documents) with origin uploaded, synthetic or promoted. The item's packet is its own logical copy (same blobs): deleting the production packet does not touch the dataset.

Editing, tags, sub-datasets, snapshots

  • PUT …/items/{item}/ground-truth with If-Match: "v<n>" creates version n + 1; the history (GET …/history) keeps every version and a stale If-Match is 412.
  • GET /v1/datasets/{id}/tags counts items per tag, with untagged.
  • POST …/subsets saves a filter (tags_all, tags_any, splits): a sub-dataset is a view.
  • POST …/snapshots freezes the items with their ground-truth version, split and tags under a content_hash (SHA-256 of the RFC 8785 form of the list); asking again for unchanged content returns the same snapshot (200). Evaluations reference snapshots, so editing a ground truth never changes a past evaluation.

The sealed test split

The optimizer, the agent and the calibration never read the items or ground truth of test (403 TEST_SPLIT_SEALED, by API, by service and by direct repository call); people read it and each read is logged (GET …/test-split-access). Sending X-DX-Access-Context: agent (or optimizer, calibration) marks a call as one of those; the header can only narrow access.

Evaluations

curl -X POST $DX/v1/evaluations -H "Authorization: Bearer $KEY" -d '{
  "alias": "facturas@dev", "snapshot_ids": ["dss_…", "dss_…"],
  "scope": {"splits": ["validation"], "tags": ["es"]}}'
  • The default scope is train + validation; test needs "confirm_test": true (422 CONFIRM_TEST_REQUIRED without it) and is logged.
  • With several snapshots the metrics are over their union; per_dataset breaks them down and an item present in two datasets is flagged duplicate_across_datasets and counted once in the union.
  • GET /v1/evaluations/{id} has metrics (T10's metrics with 95 % bootstrap intervals, per_field, per_tag), GET …/fields the expected-versus-obtained rows (filter by class, tag, path, split; keyset pages), GET …/items the per-document results and GET …/tags the per-tag accuracy. A run that failed counts as a failed document with every labelled field omitted.
  • GET /v1/evaluations:compare?a=&b= is the paired comparison of T10 (bootstrap of the difference, exact McNemar test with Holm's correction); 422 SNAPSHOTS_DIFFER if the two evaluations were not made on the same snapshots and scope.

Workflows with Classify and Split

Create the dataset with "gt_kind": "segmented" (the workflow must have a Classify or Split node, otherwise 422 EVAL_NOT_APPLICABLE) and upload ground truth shaped as {"segments": [{"id", "category", "pages": [...], "data": {...}}], "unassigned_pages": [...]}. The evaluation adds classification (accuracy, macro F1, confusion matrix with unclassified) and partition metrics (boundary F1, first-page F1, segment IoU, exact segments, perfect packets, Pk, WindowDiff) under metrics.classification and metrics.segments, and scores the fields of each segment after pairing it with the ground-truth segments (a missing segment's fields are omissions, an extra segment counts in spurious_segments). Pk and WindowDiff are unreliable when segments average fewer than 4 pages (unreliable_short_segments).