Skip to content

Health: datasets and evaluations

The two Health screens live inside a workflow (/workflows/{id}/health/…). They are the interface of datasets and evaluations: the API they use is described there and in Quality contracts; this page explains how to read what they show.

  • S9 · Datasets — /health/datasets (the list) and /health/datasets/{dataset} (the ground-truth grid, splits, tags, snapshots).
  • S10 · Evaluations — /health/evaluations (the numbered list), /evaluations/{id} (the detail) and /evaluations/compare?a=&b=.

The states of a ground-truth cell

Every cell of the grid has one of three states, written with an icon and a word (never only a colour):

state what it says counts in a metric?
Etiquetada / Labelled (check) a person wrote the expected value yes: right or wrong
Ausente / Absent (slashed circle) the document does not have this field: the expected value is null yes: any answer is a hallucination
Sin anotar / Unannotated (dashed circle, highlighted) nobody labelled it no: it is in no denominator

A table field shows N filas and expands below the grid; each of its cells has the same three states. Editing a cell (double click, Enter or F2) saves the whole ground truth as the next version with If-Match; if someone else saved first the screen keeps what you typed, says so, and offers to reload and reapply your edit on the new version. Every save is an entry of the item's history (who, when, why, content hash). Items of the test split carry the mark sellado / sealed: the optimizer, the agent and the calibration never read them, and every read by a person is logged.

Reading a figure: value, interval and n

Every number of S10 is written 0,962 [0,951–0,972] n=1.204:

  • the value is the point estimate;
  • the interval is a 95 % bootstrap interval that resamples documents (not fields: the fields of a document are not independent), so a few long documents do not make the interval look narrower than it is;
  • n is what the value is a fraction of: scored fields for the field metrics, documents for the perfect-document rate.

With few documents the interval is wide and the screen says muestra pequeña / small sample. The list has a compact view that shows only the value (the interval and n stay in the tooltip). Cost and p95 latency are not measured by an evaluation yet and show —.

Expected versus obtained

Each scored field is one row: the expected value, what the workflow answered, the class of the result and its confidence. Classes: correct, format_only (right after normalization, written differently), omission (there was a value, none answered), hallucination (the document has no such field, something was answered) and wrong_value. The table is sortable and filterable, virtualized (only the rows in view are in the DOM, ≤ 80 with 2 000 rows) and each row links to the viewer at that field when the run is known (/runs/{run}?field=…).

The error matrix (GET /v1/evaluations/{id}/error-matrix) counts, per field, every class; its errors column is everything that is not correct, the same count as metrics.per_field[].errors.

The reliability diagram, ECE and the noise floor

GET /v1/evaluations/{id}/reliability groups the fields that carry a confidence into 15 bins of equal mass (the same number of fields in each, so no bin is empty) and for each gives the mean confidence, the observed accuracy, a 95 % Clopper-Pearson interval of that accuracy and n. A perfectly reliable score lies on the diagonal. The ECE is the weighted mean distance to the diagonal.

The ECE of a finite sample is never 0, even for a perfect score. The noise floor is the ECE a perfectly calibrated evaluator would show with the same confidences and the same n (200 simulated draws, fixed seed): an ECE at or below the floor is indistinguishable from noise, one above it is a real miscalibration. The screen writes both, plus n: ECE 0,034 · suelo de ruido 0,028 · n=1.204. With fewer than ten fields per bin the screen warns n_too_small.

The confidence of the pipeline is a heuristic score, not a probability unless a calibrator (T39) produced it. The response carries scores_kind (heuristic, calibrated, none):

  • heuristic → the screen shows «Confianza heurística: no es una probabilidad» and never the word calibrated;
  • calibrated → it shows the calibrator (id, version, n_samples);
  • none (no confidence in the evaluation) → «Sin datos de calibración», not a chart of zeros.

The risk-coverage curve (…/risk-coverage, ≤ 200 points) accepts the most confident fields first and shows the error rate among the accepted ones for each coverage; its area (AURC) is descriptive. Every chart has a data table behind Ver tabla de datos.

ReliabilityDiagram (shared with T40)

web/src/components/charts/ReliabilityDiagram.tsx is the contract T40 (monitoring by version) reuses for the reliability of production:

interface ReliabilityDiagramProps {
  bins: { p_mean: number; acc: number; lo: number; hi: number; n: number }[]; // increasing confidence
  ece: number | null;
  noise_floor: number | null;
  kind: "heuristic" | "calibrated" | "none";
  calibrator?: { id?: unknown; version?: unknown; n_samples?: unknown } | null;
}

It draws with plain SVG (no chart library) and is a deferred chunk. MetricValue {value, ci95, n} formats a figure with ICU (formatMetric).

Comparing two evaluations

Only evaluations of the same snapshots and scope can be compared (the selector disables the others and a 422 SNAPSHOTS_DIFFER is shown as text). The comparison is paired by document:

  • the difference per metric (A − B) with a paired 95 % interval, and the exact McNemar test on the documents one gets perfectly right and the other does not, with Holm's correction for the family of tests (p ajustado);
  • per field (GET /v1/evaluations:compare-details): the accuracy difference with a paired bootstrap interval over documents, and the minimum detectable difference — the smallest |Δ| this comparison could have told apart from noise at 5 % significance and 80 % power (2,8 × the bootstrap standard error; for the perfect documents 2,8 × √(discordant / n) / √n). A non-significant difference smaller than the minimum detectable one means not enough data, not no difference;
  • the documents where B got worse (lower field accuracy), worst first.

Significance is a word and an arrow (▲ A better, ▼ B better, = no significant difference), never only a colour. For hallucination_rate lower is better.

Promoting a verified document

POST /v1/datasets/{id}/items:promote {run_id, tags?} turns a document reviewed in production (T32) into a ground-truth item without typing it again. The ground truth is the effective result of the review — approved and corrected fields are labelled, fields marked absent are absent — and the item keeps its own copy of the document, so deleting the production packet does not touch it.

  • 409 PROMOTION_INCOMPLETE while the document is not verified or a field is still needs_review (or rejected with no corrected value): errors[].pointer is /state, /fields/<path> or /run.
  • Idempotent per (dataset, run): promoting the same run again returns the same item (200).
  • A run whose data was deleted (certified deletion, ZDR) cannot be promoted; a run of another workflow is 400 validation-error at /run_id.