Skip to content

Encryption and the key hierarchy

Data governance (T50) sits on the envelope encryption of the data platform (T11). This page states what is encrypted, with which keys, and where the boundary of the guarantee is.

Hierarchy

KEK of the tenant         local file (HKDF per tenant) or OpenBao Transit key dx-t-<tenant>
 └─ DEK of the tenant      tenant_keys: versioned, wrapped by the KEK (T11; still used for old blobs)
 └─ DEK of one blob        data_key: AES-256, one per blob *content*, wrapped by the same KEK (T50)
      └─ object key        HKDF-SHA256(DEK, salt per object) → AES-256-GCM, 1 MiB chunks
  • Blob key. t/{tenant}/blobs/aa/bb/<sha256 of the plaintext>: one object per content and tenant (no deduplication across tenants).
  • Envelope DXE1 format 2. Header DXE1 | 2 | 0 | salt | chunk_log2 | data-key id (16 bytes). Every chunk is authenticated with the header, the tenant id, the plaintext sha256 and a last-chunk flag as AAD. An object copied to another tenant, to another blob key, truncated or reordered fails to decrypt (BlobIntegrityError). A stored object never contains %PDF or any other plaintext.
  • Format 1 (DEK per tenant version) stays readable. Objects written before T50 have no data key of their own: deleting them is a plain delete (the certificate says method: delete, not crypto_shred).
  • Derived artifacts (DocIR, page images, results) are blobs like any other and get their own data key; deleting a packet destroys the keys of every artifact of the runs that processed it.
  • Reference counts. A content that another packet (or a dataset item) still references is retained: its key is not destroyed and the certificate lists it under retained. The count is computed from the referencing tables, never from a counter that could drift.
  • Cache. Unwrapped data keys live in a per-process cache (5 minutes, 1 024 entries) that is emptied for a key as soon as it is destroyed.

Crypto shredding

Destroying a data key is erasing its wrapped bytes from Postgres (status = destroyed, wrapped_dek IS NULL, enforced by a CHECK). From then on nobody — not even with the KEK — can decrypt the object, wherever it was copied (object-store versions, replicas, exports of the bucket).

Database backups keep the wrapped key as it was when they were taken. That is why every deletion writes its signed certificate to the object store under t/{tenant}/erasure/ (outside the database) and extractit admin erasure replay --org <id> re-applies those certificates after any restore.

What the local KEK cannot do

With DX_KEY_PROVIDER=local every KEK is derived from one master key file (DX_KEK_FILE, mode 0600). Destroying a tenant KEK is not verifiable there: the master key still derives it, and backups of the file keep it. Destroying the data keys still makes the blobs unreadable, but if the master key and an old database backup both leak, the old wrapped keys can be unwrapped. For strong erasure requirements use OpenBao Transit (DX_KEY_PROVIDER=openbao): the KEK never leaves OpenBao and deleting the tenant key there is a verifiable destruction.

Boundary of the guarantee

Covered: Postgres rows, the encrypted objects of the object store (all versions and delete markers), the data keys and their in-memory cache, and the values of the review (see below).

Not covered, and not claimed by any certificate: copies held by model providers beyond the retention class listed in provider_retention; exports, downloads and webhook payloads already delivered; backups of the object store made outside the product; any storage that is not inventoried by the operator.

Review values (review_decision, field_override)

What a reviewer saw or typed is extracted data of the document, but review_decision is append-only (dx_app cannot update or delete it, and neither can dx_system). Its values are therefore sealed: previous_value, new_value, the quoted evidence and the free-text reason of every decision, and value and evidence of every field_override, are stored as AES-256-GCM envelopes ({"$sealed": 1, "kid", "n", "c"}) with a data key per run (a data_key row wrapped by the tenant KEK, like the blob keys; its sha256 is derived from the run id). The associated data binds tenant, run, table, row and column, so a ciphertext cannot be moved to another row. The rest of the row (ids, action, path, by, at, reason_code, times) stays in the clear and the table stays append-only.

The erasure saga (certified DELETE, retention TTL, ZDR) destroys that key together with the blobs of the run: the certificate lists review_decision and field_override in destroyed[] with method: crypto_shred and the dek_id, a step verifies that no active key is left, and the erasure ledger re-applies the destruction after a restore. From then on the review API reads those values as null with redacted: true (GET /v1/runs/{id}/review/history; GET /v1/runs/{id}/review of a deleted run answers the decided fields with no value). Migration 0016_t50_review seals the rows that existed before; it needs the same key provider the API uses (DX_KEY_PROVIDER, DX_KEK_FILE or DX_OPENBAO_*) whenever there are rows to seal.

Still in the clear and outside this guarantee: review_document.final_result (the snapshot of a verified review; the saga deletes the row, backups keep it until they expire), the field.corrected and review.completed outbox events already delivered, and the audit event of a review or reject that carries a reason in its metadata.

Rotating the KEK (python -m extractit.store.admin rewrap-kek) rewraps the tenant DEKs; wrapped data keys keep their KEK version and stay decryptable (Transit keeps old versions) until a rewrap of data_key is added.