Evaluation: extractit eval
extractit eval measures the pipeline on a dataset with one reproducible command: the metrics of
research/09 §5 with bootstrap intervals, a paired comparison with significance, the baseline "PDF
straight to the LLM" and a gate for CI.
extractit eval --dataset bench/datasets/invoices-mini --model oracle # no LLM, deterministic
extractit eval --dataset bench/datasets/invoices-mini --model codex --baseline # system vs baseline
extractit eval --dataset bench/datasets/invoices-mini --model oracle --fail-under 0.97
extractit eval compare reports/eval-A.json reports/eval-B.json
It writes eval-<run_id>.json (contract schemas/eval-report-1.0.json) and eval-<run_id>.md
into --out (default reports/) and prints a summary. Exit codes: 0 done, 1 the metric is
below --fail-under, 2 incorrect usage or a dataset whose files do not match its manifest.
Dataset format
A dataset is a directory:
manifest.json name, version, schema{file, sha256}, generator, items[...], dataset_sha256
schema.json the extraction schema (with field ids)
docs/<id>.pdf the documents
gt/<id>.json the ground truth of each one
items[] has {id, doc, gt, sha256_doc, sha256_gt, tags, origin, pages}. dataset_sha256 covers
the ids, the document and ground-truth hashes and the schema hash: change any byte and extractit eval
refuses to run (exit 2, naming the file). The contract is schemas/eval-manifest-1.0.json.
The ground truth of a document is {data, gt_field_status, raw_text, boxes}:
data: the expected object (typed values, ISO dates), like the extraction output'sdata.gt_field_status[path]:labelled,absent(the document does not contain it: any value is a hallucination) orunlabelled(nobody labelled it: it never counts).raw_text[path]: the value as printed.boxes[path]:[{page, bbox}], normalized to the page (0 <= x0 < x1 <= 1, origin top-left), one box per line.
The synthetic generator
bench/generators/invoices.py writes invoices in Spanish, English, French and German from three
Jinja2 templates, rendered to PDF with WeasyPrint. Every value sits in <span data-gt="path">; the
box of the value is read from WeasyPrint's box tree (one fragment per line) and its vertical extent
is snapped to the glyph extent of the PDF, which is the rectangle a parser reports. A quarter of the
invoices omits the purchase order and one more field (ground truth absent). The output depends
only on the seed.
uv run python bench/generators/invoices.py --n 20 --seed 7 --out bench/datasets/invoices-mini
uv run python bench/generators/invoices.py --n 20 --seed 7 --degrade scan,rotate,photo --out ...
--degrade writes degraded/ with image-only PDFs (scan, rotation, photo) and the ground-truth
boxes transformed by the same projective map as the image. They carry the tag requires_ocr: until
OCR (T05) is integrated extractit eval omits them with a warning. WeasyPrint needs Pango and Cairo
(Homebrew: brew install pango cairo libffi; the generator sets DYLD_FALLBACK_LIBRARY_PATH on
macOS). Generation is a development task: evaluating a dataset needs no WeasyPrint.
Models
--model |
What it is |
|---|---|
oracle |
Answers from the ground truth of the document in turn: it finds the printed text in the anchored markdown of the request and returns value, anchors and quote (or not_found for absent fields). It measures parse + verification + confidence without an LLM. Deterministic. |
fake |
The oracle with a seeded noise (7 % wrong values, 4 % omissions, 30 % hallucinated absent fields; 14 %, 6 % and 50 % for the baseline). The noise of a field depends only on seed, document and path. It is a simulation, not a measurement of any model. |
fake:<dir> |
Replays cassettes (T03). |
codex[:model], gemini[:alias] |
Real backends (extractit doctor). |
--tier fast|standard|agentic is passed to the pipeline. --repeat N runs N times and the report
adds the variance of the accuracy; the metrics are those of the first run. --limit N takes the
first N documents, --jobs 2 evaluates two at a time, --seed fixes the bootstrap and fake.
Metrics (research/09 §5)
All intervals are 95 % percentile intervals of a bootstrap that resamples documents (10 000
replicas by default; AUROC 1 000). The metrics section depends only on the dataset, the answers
and the seed: repeating a run gives the same bytes. Wall-clock numbers are in operation.
- Fields (§5.1). After normalizing (dates to ISO, numbers within ±0.005 or 1e-6 relative,
strings NFKC with collapsed spaces and lower case unless
normalize.case = "none"), each labelled or absent field is TP, FP+FN (wrong value), FN (omission), FP (hallucination, absent field) or TN. Accuracy(TP+TN)/N, precision, recall, F1; hallucination rate (hallucinated / absent fields); correct abstention (TN / nulls); coverage (non-null / N); ANLS for strings. Table cells are measured as tables, not here. - Perfect document. Every field and every table row right. Reported by page range (1–5, 6–15, 16–50, 51+) with Wilson intervals.
- Tables (§5.2). Hungarian matching of rows by the
x-dx.table.row_key(or the whole row): row recall and precision, duplicates, Kendall τ of the order and cell accuracy over matched rows. - Citations (§5.4). Over the literal fields (labelled, with a ground-truth box): page
accuracy, IoU between the union of the cited regions and the box, share with IoU ≥ 0.5 and mean
IoU, word precision (cited words that lie in the box),
verifiedrate and verified-and-correct rate. A missing prediction scores IoU 0, so abstaining cannot raise the numbers. - Calibration (§5.5), informative. AUROC of correctness, AURC, AUGRC, precision at 80/90/95 % coverage, ECE with 15 equal-width and 15 equal-mass bins, Brier and skill score, log-loss, calibration slope (IRLS), reliability tables, error by tail band and the noise floor (the ECE a perfectly calibrated evaluator would show with the same N). The confidence of the pipeline is a heuristic: this describes it and gates nothing (calibration proper is T39).
- Operation (§5.6). Latency p50/p95/p99, tokens, cost per document and per 1 000 pages. Cost is
n/afor subscriptions, agents and simulated models; it is never estimated from list prices.
A document that fails counts as an error with all its fields as omissions and is listed in the
report. With the baseline the report has both columns; the baseline sends the page images with the
same field guide but the output schema {value, status, page} (plus bbox with
--baseline-bbox), no anchors, no verifier; its IoU is n/a without boxes.
Comparing two runs
extractit eval compare A.json B.json [--bootstrap 10000] [--alpha 0.05] pairs the documents and
reports A − B for field accuracy, F1, hallucination rate, citations with IoU ≥ 0.5 and perfect
documents, each with a paired bootstrap interval (documents, then fields inside each), and an exact
McNemar test on perfect document and its three partial versions (all fields, all tables, all
citations) with Holm's correction over that family. If the reports cover different documents it
compares the intersection and says so. --system-a/--system-b baseline compares the system with
the baseline of one report.
The gate in CI
The job eval-gate of ci.yml runs the oracle over invoices-mini and fails below 97 % field
accuracy (--fail-under 0.97), 97 % verified literal fields (--metric verified_rate) or 90 %
of citations with IoU ≥ 0.5 (--metric iou_ge_05 --fail-under 0.90). --metric also accepts f1.
A regression of the verifier makes the second one exit 1 (see tests/eval/test_cli_eval.py).
Limits
Degraded variants: their ground-truth boxes are in the frame of the image, while the parser puts
its boxes in the frame it corrected (rotation, deskew), so citation IoU on rotate and photo
variants is not comparable (0 % in the first measurement, bench/reports/eval-ev_8029*.md); field
values are. Mapping the boxes through rotation_applied and deskew_angle_deg is open work.
The synthetic ground truth overestimates real quality: results are reported by origin and tag, and
DX-Bench (T52) adds public documents. Scans need OCR (T05). The simulated models measure the
harness, not a model. Managed datasets and sealed splits are T38.