Air-gapped install (commercial edition)
The air-gapped profile installs ExtractIt on a cluster without any network path to the
internet: local models (vLLM on GPU, or llama.cpp on CPU), telemetry off for good, every egress
outside the cluster denied, and a signed bundle verified file by file before anything is loaded.
It is part of the commercial edition (ee/, FSL-1.1-ALv2); the Apache-2.0 chart of
the standard install is unchanged and installs on its own.
| Piece | Where |
|---|---|
| Bundle build, verify, install | ee/deploy/airgap/ (build-bundle.sh, verify.sh, install.sh) |
| Chart and profiles | ee/deploy/helm/extractit-ee/ (values-airgapped.yaml, values-airgapped-cpu.yaml) |
| Models | ee/tools/airgap/models.yaml → models.lock.json (mirror_models.py) |
| Tools in the image | python -m extractit_ee.airgap … (also extractit airgap …) |
| CI | .github/workflows/airgap-install.yml |
What the bundle contains
dx-airgap-<version>.tar.zst is a zstd tar of one directory:
| File | Content |
|---|---|
haul.tar.zst |
hauler store save: every image of images.yaml (T53) and images-ee.yaml by digest, both charts, and the model weights as Hauler files (<model>__<path>) |
models.lock.json |
per model: name, pinned revision, role (llm / vlm), served name, SPDX license, every file with sha256 and size |
charts/ |
extractit-<v>.tgz (Apache-2.0) and extractit-ee-<v>.tgz |
values/ |
values-airgapped.yaml, values-airgapped-cpu.yaml |
sbom/, signatures/ |
T53's CycloneDX SBOMs and cosign signatures of the release |
images.yaml, images-ee.yaml, images.txt, hauler-manifest.yaml |
what went into the haul |
bundle.manifest.json + .sig |
sha256 and size of every file above, signed with cosign sign-blob (offline release key) |
The release public key (cosign.pub) and the verify kit (verify.sh, dx_airgap_verify.py,
install.sh) travel out of band next to the bundle: never trust the copies inside it.
1. Build the bundle (online, at release)
export COSIGN_KEY=release.key COSIGN_PASSWORD=… # offline release key
export SBOM_DIR=release/sbom SIGNATURES_DIR=release/signatures
export EXTRA_IMAGES=operators.txt # CloudNativePG operator, NVIDIA device plugin…
ee/deploy/airgap/build-bundle.sh 1.4.0 # -> dist/airgap/dx-airgap-1.4.0.tar.zst
Order of the build: models-verify --lock-only (licenses), mirror_models.py download (weights
at the pinned revisions, sha256 checked in streaming), models-verify again on the files, both
charts packaged, hauler store add image|chart|file, hauler store save, bundle.manifest.json,
cosign sign-blob, tar --zstd. MODELS_PROFILE=cpu builds a small bundle with the CPU model
only.
2. Verify (offline, before loading anything)
./verify.sh dx-airgap-1.4.0.tar.zst --key cosign.pub # exit 0 verified, 1 failed, 2 usage
The archive is extracted into a private directory, the manifest signature is checked with the release key (no transparency log: offline), then every file's sha256 and size is recomputed in 1 MiB chunks (a 16 GB weight is never held in memory). A file the manifest does not list fails too. One altered byte in an image layer, a weight, a chart or the manifest returns 1 and nothing is loaded.
3. Install
./install.sh dx-airgap-1.4.0.tar.zst --key cosign.pub --profile gpu \
--values site.yaml # StorageClass, publicUrl, ingress…
install.sh runs verify.sh, then hauler store load, serves the store on this host
(hauler store serve registry on :5000 and fileserver on :8080, or uses --registry /
--weights-url), and installs the ee chart with global.imageRegistry pointing at the local
registry, models.lockJson and --wait (15 min). Before any workload starts, two hook Jobs run:
- guard (
extractit airgap check-endpoints): fails the install withexternal_endpoint_forbiddenif any endpoint of the configuration leaves the cluster. The same rule is applied at render time (helm templatefails) and by the API's start-up hook when the ee image runs it. - preflight: exit codes are a contract.
| Code | Meaning |
|---|---|
| 0 | StorageClass, memory, GPU and local registry OK |
| 2 | configuration error (no cluster credentials) |
| 10 | the StorageClass is missing (or no default one) |
| 11 | no schedulable node has the minimum allocatable memory (preflight.minMemory) |
| 12 | gpu.enabled but no node has an allocatable nvidia.com/gpu |
| 13 | the local registry does not answer GET /v2/ |
Profiles
GPU (values-airgapped.yaml) |
CPU degraded (values-airgapped-cpu.yaml) |
|
|---|---|---|
| Extraction LLM | vLLM, qwen3-8b (Apache-2.0) |
llama.cpp, qwen3-1.7b-q8 GGUF (Apache-2.0) |
| Document VLM (route C of T20) | vLLM, qwen3-vl-8b (Apache-2.0) |
none: the router keeps to paths A/B |
| Scheduling | nvidia.com/gpu: 1, toleration and nodeSelector per server |
no GPU (the chart refuses gpu.enabled here) |
Both profiles share the bundle. The core reaches the models through one OpenAI-compatible endpoint,
dx-models (DX_COMPAT_BASE_URL=http://dx-models:8000/v1), which routes by model name to
dx-vllm-llm / dx-vllm-vlm / dx-llamacpp. Model servers run with HF_HUB_OFFLINE=1,
TRANSFORMERS_OFFLINE=1, usage statistics off and xgrammar guided decoding; their init container
(extractit airgap models-fetch) copies the weights from the in-cluster file server and recomputes every
sha256 before the server starts. The platform model policy (/etc/extractit/airgap/model-policy.toml)
sets air_gapped = true: T15's resolver accepts only local routes. In the CPU profile the only VLM
engine of the catalogue is refused by that policy, so a workflow that asks for a VLM tier still
runs on A/B (warning no_eligible_engine).
Models are chosen in ee/tools/airgap/models.yaml and measured with T10's harness; change them
there (then mirror_models.py lock), never in code. Only Apache-2.0, MIT and BSD weights are
mirrored: AGPL/GPL, unknown or any other license is refused by mirror_models.py, by
models-verify and by the chart.
Confidence. Local models have no calibrator until T39 fits one on their snapshot, so scores
stay confidence.kind = heuristic. A calibrator fitted on another model_snapshot is never loaded
(calibrator_snapshot_mismatch).
Telemetry and support
values.schema.json of the ee chart rejects, with airgapped.enabled=true:
telemetry.enabled=true, observability.mode other than none, a Gatus page, a Langfuse or OTLP
endpoint, OTEL_SDK_DISABLED other than true, HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE other than
1, network policies off, and any egress CIDR that is not private. No Collector, no error sink.
A support archive exists only when an operator asks for it:
kubectl -n extractit logs deploy/dx-worker --since=24h > logs/worker.log
python -m extractit_ee.airgap support-bundle --logs logs/ -o support.tar.gz
Only structured log lines are kept, with an allow list of keys (event, level, ids, codes, timings); document text, prompts, model answers and secrets are dropped, remaining strings are scrubbed (tokens, keys, credentials in URLs, e-mails, IBANs) and free-text lines are omitted.
Upgrades: the N-2 rule
An air-gapped release upgrades in place from the two previous minor versions of the same major.
python -m extractit_ee.airgap upgrade-check --from 1.2.0 --to 1.4.0 # allowed: prints the steps
python -m extractit_ee.airgap upgrade-check --from 1.0.0 --to 1.4.0 # refused (exit 1): go to 1.2.x first
Steps: back up (CloudNativePG backup and the dx-home Secret) → verify.sh the new bundle →
hauler store load and serve → preflight → helm upgrade with the profile values (the migrations
Job expands the schema first; never a downgrade) → models-verify and the smoke extraction → keep
the old bundle until the smoke test passes (rolling back means restoring the backup).
Quality against the cloud
On the GPU test machine, run T10's harness on its synthetic dataset (≥ 200 documents, fixed seed) with the local models and with the cloud reference, same documents and schemas, then:
python -m extractit_ee.airgap quality-gate --local local.json --cloud cloud.json -o parity.md
It pairs the two reports with T10's comparison (hierarchical paired bootstrap) and passes when
|ΔF1| ≤ 1.0 point, at least 200 documents are paired and the run recorded no external connection
(run the eval inside SocketGuard or in the air-gapped cluster). If the gap is larger, the cause is
documented and the model decided before closing.
Hardware matrix
| Profile | Hardware | Driver / runtime | Memory | Status |
|---|---|---|---|---|
| GPU | 1× NVIDIA GPU with ≥ 24 GB per model server (L4, A10G, L40S, A100) | NVIDIA driver ≥ 550 with the precompiled driver container and device plugin mirrored | 64 GB node | not yet measured: pending the GPU test machine |
| CPU degraded | 8 vCPU | — | 16 GB node | chart and fake-model path tested in CI; real llama.cpp run pending |
GPU images and drivers must come from a precompiled driver container or your own mirror (add them
to EXTRA_IMAGES). Certify a row only after the smoke test and the quality gate pass on it.
ADR-017 addendum: bundle format
- Hauler (Apache-2.0) is the bundle tool:
hauler store add image|chart|file,hauler store saveand, offline,hauler store loadplushauler store serve registry|fileserver. - Verified on day one (Hauler 1.2.5): local charts are added by absolute path; files are stored
flat (we name weights
<model>__<path>); the served registry exposes repositories without the source registry host (soglobal.imageRegistryrewriting works). Finding: an image added with--platformfrom a multi-arch index is stored under the index digest but only one platform manifest, andserve registrythen rejects it (DIGEST_INVALID); the build therefore keeps all platforms by default (PLATFORM=empty), since the charts pin index digests. - Not yet verified: files of tens of GB (the lock holds 16–18 GB per GPU model). If Hauler's
file store does not cope on the build machine, the fallback is Zarf (Apache-2.0) for the
weights, keeping the same signed manifest and
verify.sh. - Two layers on purpose: Hauler's archive has no place for an external signed manifest, so the
outer tar holds the haul plus
bundle.manifest.jsonsigned withcosign sign-blob. - No offline license file in this scope (and no Zarf package for defence/government).
CI: airgap-install
.github/workflows/airgap-install.yml (release tags and on demand) builds the bundle, creates an
internal: true Docker network, a canary DNS (ee/deploy/airgap/ci/canary_dns.py, the node's
only resolver: logs every query and answers NXDOMAIN), a kind cluster on that network with an
iptables audit chain counting packets to public addresses, runs install.sh --fake-llm, the smoke
extractit FILE of a sample invoice and fails on any external DNS query, any public packet, a pod not
Ready or an install over 15 min. A host on an internal: true network has no default route,
so a packet to a public address dies in the routing decision before any iptables chain could
count it: the job therefore routes the node through the gateway (the audit chain counts and drops
everything public, and the host drops it again) and, after measuring, makes one deliberate
connection from the node that must move the counter; otherwise the 0 proves nothing and the job
fails. ee/deploy/airgap/ci/network-rehearsal.sh exercises the same isolation with Docker only
and the fake model (no kind, no release images): a structured extraction call through
dx-models must leave 0 external DNS queries and 0 packets to public addresses; then a deliberate
lookup of api.openai.com must show up in the canary log and a deliberate connection with a
forced default route must be counted by the audit rule and never connect. Its last run
is kept in ee/deploy/airgap/ci/evidence/.