Model backends
ExtractIt talks to language models through one port, generate_structured: you give it a
conversation (text and images) and a JSON Schema, and you get back a JSON document that validates
against that schema, together with token usage, cost and latency. Which model answers is
configuration, not code:
export DX_MODEL=gemini:standard # Gemini API key
export DX_MODEL=openai:standard # OpenAI API key
export DX_MODEL=anthropic:fast # Anthropic API key
export DX_MODEL=compat:qwen2.5-vl # any OpenAI-compatible server (vLLM, llama.cpp...)
export DX_MODEL=codex # your ChatGPT subscription, through the official Codex CLI
export DX_MODEL=fake # offline, deterministic, for tests
DX_MODEL is gemini|openai|anthropic[:<alias|snapshot>], compat:<model>, codex[:<model>] or
fake[:<cassette directory>].
If it is not set and a Gemini key is present, gemini:standard is used; otherwise the command
fails with the list of options. Run extractit doctor to see what is usable right now.
What every backend guarantees
- Valid output. The schema is first down-levelled to the subset the provider accepts (what was
lost is reported in
StructuredResponse.downleveled). The answer is then validated against your original schema (JSON Schema Draft 2020-12). If it does not validate, the gateway asks the model to fix it, quoting the JSON Pointer of the error, at most 2 times; after that you get aSchemaViolationErrorcarrying the pointer. - Retries. Up to 3 retries on 429, 5xx and timeouts, with exponential backoff (base 1 s,
factor 2), full jitter and respect for
Retry-After. - Accounting. Every call produces a
ModelCall(tokens, latency, snapshot, purpose, prompt hash, cost) in the call ledger, which also sums perpurpose. The ledger stores hashes and counters, never document text. Logs never contain document text or keys. - Typed errors.
RateLimitedError(withretry_after),SchemaViolationError,TruncatedOutputError,ModelRefusedError,ModelUnavailableError,BackendNotInstalledError,BackendAuthError,InvalidRequestErrorandCassetteMissingError, all subclasses ofGatewayErrorwith a coarsekind(input,model,rate_limit,auth,internal). - Images up to 20 MB each.
Gemini (API key)
export DX_GEMINI_API_KEY=... # also read: GEMINI_API_KEY, GOOGLE_API_KEY
export DX_MODEL=gemini:fast # fast | standard | strong | an exact snapshot
Built on the official google-genai SDK (response_mime_type="application/json" plus
response_json_schema). Aliases map to snapshots in src/extractit/llm/models.toml, where every
entry carries its price per million tokens, the date it was verified and the source URL. The
snapshot the service actually used (model_version) is what you get in model_snapshot. Billing is
per token, so cost_usd is computed from that file (cache-aware).
Privacy: free tier
On the free tier of the Gemini API, Google may use the content you send to improve its products. For sensitive documents use a paid key. The tier of a key cannot be detected, so this notice is shown the first time you use the backend, always.
OpenAI, Anthropic and OpenAI-compatible servers
Three more adapters sit behind the same port and pass the same contract suite. Their models, regions, retention and prices live in the model registry.
export DX_OPENAI_API_KEY=... # only DX_* variables are read: a stray OPENAI_API_KEY is ignored
export DX_ANTHROPIC_API_KEY=...
export DX_COMPAT_BASE_URL=http://localhost:8000 # + DX_COMPAT_API_KEY if the server wants one
export DX_MODEL_POLICY=policy.toml # optional: see "Model policies"
| Backend | API | How the schema is enforced |
|---|---|---|
openai |
Responses API | text.format of type json_schema, strict: true, schema compiled with the openai_strict profile (closed objects, every property required, optional ones nullable, unsupported keywords dropped) |
anthropic |
Messages API | a forced tool emit_result whose input_schema is the compiled schema (anthropic_tool profile) for snapshots that accept it (Haiku 5.5), or output_config.format for those that reject a forced tool_choice (Sonnet, Opus, Fable): the registry says which |
compat:<model> |
Chat Completions | response_format with the schema if the probe sees it working, otherwise JSON by prompt (structured_output='emulated') |
In every case the answer is validated against your original schema and repaired at most twice.
PDFs (PdfPart) are sent natively where the registry says native_pdf; images where vision.
Log-probs are requested only where the registry declares them and the endpoint returns them
(StructuredResponse.token_logprob_mean is None otherwise). The SDKs' own retries are off: the
gateway's retry policy (3 retries, backoff, Retry-After) owns them, so a fallback sees one clean
failure. Documents are placed in user turns only, never in the system prompt, which also tells the
model that document text is data and never instructions.
OpenAI-compatible servers. probe(base_url, model) asks the endpoint instead of assuming:
GET /v1/models; a minimal request with a json_schema response_format (native only if the
answer really is that JSON, else a plain request decides emulated or none); a 1x1-pixel image
(vision); and a request with logprobs=true. The result is cached per (base_url, model), runs
automatically on first use, and can be added to the registry as a local route
(locality='local', retention_class='local', zdr=True) with
ModelCapabilities.to_entry(). billing='self-hosted' (default) costs 0.0; give a hosted clone
billing='api' and prices to bill it per token.
Model policies, fallback and the registry
from extractit.llm.registry import (
ModelPolicy,
Needs,
merge_policies,
resolve,
load_registry,
route_gateway,
)
policy = merge_policies([platform, organisation, project, node]) # can only tighten
route = resolve(policy, Needs(vision=True), load_registry()) # pure and deterministic
gateway = route_gateway(route) # primary + compliant fallbacks
- Policy (
ModelPolicy):allowed_providers,eu_only,zdr_required,max_usd_per_1k,air_gapped,preference,fallbacks. Merging intersects lists, OR-s the switches and takes the lowest budget, so a node can never relax what the organisation set. The effective policy has apolicy_hash(SHA-256 of its RFC 8785 JSON).DX_MODEL_POLICYnames a TOML or JSON file. - Resolution filters the registry (
provider_not_allowed,not_local,region_not_eu,zdr_unavailable,capability_missing,over_budget, in that order) and ranks the rest by(index in preference, input + output price, id): no randomness. If nothing complies it raisesNoCompliantModelErrorcarrying every rejected candidate with its reason: it fails closed. - Fallback (
FallbackGateway): walks the route and its compliant fallbacks onrate_limited,provider_unavailableandtimeout(fallback_onis configurable;schema_violationandcontent_filtereddo not fall back by default). A per-provider circuit breaker opens after 5 consecutive failures in 60 s for 30 s. If every candidate fails the call raisesProviderUnavailableError: it never improvises a provider the policy did not return. - Ledger: each logical call writes one row with
policy_hash, the route used, the rejected candidates andfallback_attempts(provider, model, outcome, latency). Metadata only: no keys, no content.
Codex CLI (ChatGPT subscription)
codex login # once; ExtractIt never reads or stores your credentials
export DX_MODEL=codex # or codex:<model> to name a model
The adapter runs codex exec as a child process with your own ChatGPT session:
codex exec --ephemeral --skip-git-repo-check --sandbox read-only \
--output-schema <tmp>/schema.json -o <tmp>/out.json --json -
- The prompt goes through stdin; images are written to the temporary directory and passed with
--image. - The working directory is an empty temporary directory, deleted afterwards. The sandbox is read-only and the session is ephemeral (nothing is written to your Codex history).
- The child gets a minimal environment (
PATH,HOME,CODEX_HOME,LANG,TMPDIR).OPENAI_API_KEYandCODEX_API_KEYare removed so that usage is billed to your plan and not to an API key, unless you setDX_CODEX_ALLOW_API_KEY=1. - At most 2
codexprocesses run at the same time in a process; each call has a 300 s timeout and the whole process group is killed when it expires. model_snapshotis the output ofcodex --version. There are no log-probs with this backend, andcost_usdisNone(the plan is not billed per token).
Limits and failure modes:
- Plan limits. ChatGPT plans have usage windows (about every 5 hours, plus a weekly one). When
they run out you get
RateLimitedErrorimmediately (no retries: waiting seconds does not refill a window), withretry_afterwhen Codex says it, and, only if a Gemini key is configured, afallback_hintofDX_MODEL=gemini:<alias>. - Latency is higher than an API call; that is why concurrency is capped.
- No session (
codex login status!= 0) givesBackendAuthErrorwith the action to take; a missing executable givesBackendNotInstalledError. - Local, personal use only. Do not put this backend behind a hosted multi-user service.
Privacy: ChatGPT account settings
With a ChatGPT login, your documents follow your account's data settings. On personal plans, turn off Improve the model for everyone before processing sensitive documents. The notice is shown once.
Both notices go to stderr (even without a TTY), once per backend; the state lives in
$DX_HOME/state.json (default ~/.extractit/). DX_PRIVACY_ACK=1 silences them.
FakeModel (tests)
DX_MODEL=fake[:<directory>] replays cassettes and needs no network:
- Key: sha256 of the canonical JSON of (model, system, messages with each image replaced by its
sha256, schema, temperature, purpose). File:
<directory>/<purpose>/<first 16 hex>.json, holding a summary of the request (sizes and hashes, no text), the answer and a fixed usage. - Missing cassette:
CassetteMissingErrorwith the full hash and the summary. FakeModel("record", inner=<real gateway>)writes cassettes and needsDX_LIVE=1. The smoke tool does it for you:
DX_LIVE=1 DX_MODEL=codex python -m extractit.llm.smoke \
--schema tests/fixtures/llm/invoice.schema.json --text-file tests/fixtures/invoice.txt \
--record tests/cassettes/llm
FakeModel.scripted([...])returns programmed answers in order (JSON, dicts, exceptions or callables) andFakeModel.stochastic(truth, p=0.8, seed=0)is right with probabilitypand otherwise changes one value, deterministically for a given seed.
Record cassettes only with synthetic documents, and check them for secrets before committing:
grep -rIl -E 'AIza|key=|x-goog-api-key|sk-' tests/cassettes must print nothing.
Checking a live backend
DX_LIVE=1 DX_MODEL=codex python -m extractit.llm.smoke \
--schema tests/fixtures/llm/invoice.schema.json --text-file tests/fixtures/invoice.txt
DX_LIVE=1 pytest -m live # live tests; excluded from CI, they spend quota or money
tests/llm/test_live_providers.py holds the OpenAI, Anthropic and OpenAI-compatible live checks
(one call each, synthetic invoice): DX_LIVE=1 DX_OPENAI_API_KEY=... pytest
tests/llm/test_live_providers.py -k openai, and likewise for anthropic and compat (which also
needs DX_COMPAT_BASE_URL and DX_COMPAT_MODEL). They were written but not run in T15: the
cassettes under tests/cassettes/{openai,anthropic,compat}/ were written by hand from each
provider's public documentation and must be compared with a real response before the adapters are
announced as validated.
Real backends refuse to run without DX_LIVE=1.