Calibrated confidence, certified thresholds and the audit
T08 gives every field a heuristic confidence (kind: heuristic): a product of signals, useful to
rank fields but not a probability. T39 (extractit.quality.calibration) turns it into a
calibrated probability per segment, certifies auto-approval thresholds with a statistical
guarantee and audits at random a share of what was auto-approved. The flow is:
- Fit calibrators from completed evaluations (T38).
- Activate the ones that passed the quality gate.
- Certify a threshold policy with the active calibrators.
- Decide each document (auto-approve or review) and audit a random share.
Recalibrate after changing the model or the prompt
A calibrator belongs to one model snapshot and one prompt hash. After a model upgrade or any
change of the prompt (schema, instructions), the next run keeps the heuristic score with
reason: calibrator_stale, and every threshold policy that used those calibrators becomes
invalidated the first time it is resolved. Run a new evaluation, fit, activate and certify
again.
1. Fit
POST /v1/calibrators:fit
{"workflow_id": "wf_…", "evaluation_ids": ["evr_…"],
"model_snapshot": "gemini-2.5-flash-001", "prompt_hash": "…"}
- Labels come from
evaluation_field_results(T38), read as thecalibrationaccess context: the sealedtestsplit is never read (403TEST_SPLIT_SEALED). A field is right when its class iscorrectorformat_only; a rightnullis right too; unlabelled fields and fields without a heuristic score are left out. - Rows of
trainfit; rows ofvalidation(other documents) gate. If a document is intrainof one evaluation and invalidationof another (a resplit), it is dropped from validation (excluded_overlap). - One calibrator per segment: each field type with enough data, plus the global segment
*of the snapshot (the fallback of a field type without its own calibrator).
| training data of the segment | method |
|---|---|
| fewer than 300 fields or 30 errors | none: the score stays heuristic (skipped, insufficient_data) |
| 300–2 999 fields, or fewer than 200 errors | Platt (logistic regression on logit(score)) |
| 3 000 fields or more and 200 errors or more | isotonic (out_of_bounds = clip) |
Quality gate on validation (15 bins): the adaptive ECE (equal-mass bins) must be at most the
noise floor + 0.01 and, with 1 000 fields or more, at most 0.03 with an AUROC of at least 0.85.
A calibrator that fails is rejected; below 1 000 validation fields it is provisional.
Each calibrator (cal_…) stores its scope, method, n_samples, n_errors, ece,
adaptive_ece, noise_floor, auroc, model snapshot, prompt hash, fitted_at and the sha256 of
its artifact: plain JSON (Platt coefficients or isotonic thresholds) in the encrypted blob store.
Nothing is pickled; the loader accepts only that JSON.
2. Activate
POST /v1/calibrators/{id}:activate makes a fitted calibrator the active one of its segment and
archives the previous one (:archive retires one by hand). In code, the hook of T08's pipeline is:
from extractit.quality.calibration.registry import CalibratorRegistry
from extractit.extract.pipeline import PipelineOptions, extract
hook = CalibratorRegistry(session, tenant_id, project_id, artifacts=store).confidence_post(wf_id)
out = extract(pdf, schema, options=PipelineOptions(confidence_post=hook))
A field then reports kind: calibrated_probability with
calibrator {id, version, scope {field_type, model_snapshot}, ece, n_samples, fitted_at} — only
when the run's model snapshot and prompt hash are the calibrator's. The document confidence stays
heuristic.
3. Certify
POST /v1/threshold-policies:certify
{"workflow_id": "wf_…", "evaluation_ids": ["evr_…"]}
For each field type, Learn-then-Test certifies the lowest threshold λ whose selective error
(wrong fields among those with probability ≥ λ) is at most α with probability 1 − δ:
- fixed grid of 52 thresholds: 0.50 to 0.99 in steps of 0.01, then 0.995 and 0.999; its sha256 is
the policy's
grid_hash(a different grid is refused withGRID_MISMATCH); - exact binomial p-value
P(Bin(n, α) ≤ k)on thenaccepted fields ofvalidationwithkerrors (withα = 0.01and no errors, 230 accepted fields certify, 229 do not); - fixed sequence: the start is the lowest
λwhose risk ontrainis at mostα / 2; the test walks to laxer thresholds and stops at the firstp > δ(the last success is certified).method: bonferronicertifies everyλwithp ≤ δ / 52instead; α= 0.01 for amounts (number,integer) and 0.03 for the rest (sendalphaper type to tighten identifiers, which are strings),δ= 0.10, at least 1 000 certification fields per type;- final threshold =
max(λ_cost, λ_LTT), with the cost thresholdp* = 1 − C_R / ((1 − e_H) · C_E)(review cost 0.30, cost of an error 30, human error 0.05: 0.9895).
| error | when |
|---|---|
422 NOT_CALIBRATED |
no active calibrator: the scores are heuristic; nothing is created |
422 CERT_SET_OVERLAPS_CALIBRATOR |
the certification documents were used to fit a calibrator |
422 INSUFFICIENT_DATA |
fewer than 1 000 fields in every field type (n_required) |
422 NO_CERTIFIABLE_THRESHOLD |
no threshold of the grid passes for any field type |
403 TEST_SPLIT_SEALED |
split: test |
The policy (thp_…) is immutable (a database trigger): per field type lambda, alpha,
delta, n_cert, n_accepted, n_errors, upper_bound (Clopper-Pearson), coverage, the
calibrators and evaluations used and a reference_histogram (20 bins of the calibrated scores of
the certification set, for drift monitoring in T40). :tighten moves every threshold one grid step
up and sets needs_recertification; only a new certification relaxes it. :invalidate retires it;
a change of model snapshot, prompt or calibrator invalidates it when it is resolved for a run.
4. Decide and audit
decision = ThresholdService(session, tenant_id, project_id, actor=..., artifacts=store).decide(
result, validations, policy_id
) # AutoApprovalDecision {auto_approved, reasons[], field_decisions[]}
A document is auto-approved when every required field has a calibrated probability at least
the threshold of its type, no blocking rule failed and the policy is valid. Without required
fields in the schema, every field with a value counts. The reasons say why not
(required_below_threshold, required_not_calibrated, required_missing,
field_type_not_certified, blocking_rule_failed, policy_invalid, no_fields).
The audit (AuditSampler.consider(run_id, decision), or POST /v1/audit-samples:consider) selects
an auto-approved run when HMAC-SHA256(seed, run_id) / 2^256 < π (2 % by default, 0.5–10 %): the
same seed gives the same selection, and without it the selection cannot be predicted. The seed of
each workflow is 32 random bytes encrypted with the tenant's key (T11 envelope); it never appears
in the API or in logs, and rotating it starts a new seed_version. Every considered run is stored
with its stratum, pi and selected; a selected run gets an audit.sampled event in its outbox
and is handed to the AuditTaskSink (the review inbox of T33).