Skip to content

API conventions

Every endpoint of the ExtractIt REST API follows the conventions below. They are implemented once, in src/extractit/api/ (T12); a resource only declares what it needs. The contract is openapi/openapi.json (OpenAPI 3.1), protected in CI by Spectral and oasdiff.

The examples use $DX for the base URL and $DX_API_KEY for a key.

Authentication and permissions

Requests carry a project API key: Authorization: Bearer dx_<env>_<key_id>_<secret>.

curl -s "$DX/v1/me" -H "Authorization: Bearer $DX_API_KEY"
  • Keys belong to a project. env is live or test; key_id (12 base32 characters) is public; the 43-character secret is shown once, at creation, and only its HMAC-SHA256 (with a versioned pepper) is stored. The pattern for secret scanners is docs/security/key-pattern.txt.
  • A key has scopes (read, write, review, eval, admin) and a role cap: the role of whoever created it. Effective permissions = permissions of the scopes ∩ permissions of the role. The matrix of the six roles and 17 permissions is docs/security/rbac.md (generated).
  • Keys expire (90 days by default, at most 365), rotate with an overlap (24 h by default, at most 168 h) and are revoked with DELETE /v1/api-keys/{id}; a revocation reaches every replica in at most 5 seconds. A project has at most 20 active keys.
  • Missing or bad credentials: 401 with WWW-Authenticate: Bearer; the code says missing-credentials, invalid-key, key-expired or key-revoked (the last two only for the holder of the secret). Scope or role too low: 403 insufficient-scope or permission-denied. After 10 failed authentications in a minute, an address gets 429.
  • Resources of another organization or project are always 404, never 403.

The first key of an organization comes from the bootstrap command:

python -m extractit.api.admin create-api-key --tenant org_… --role owner --scopes admin

Errors (RFC 9457)

Every 4xx and 5xx is application/problem+json:

{
  "type": "https://docs.extractit.dev/errors/validation-error",
  "title": "Request validation failed",
  "status": 400,
  "code": "validation-error",
  "retryable": false,
  "request_id": "req_01jc…",
  "errors": [{"pointer": "/slug", "rule": "string_pattern_mismatch", "message": "…"}],
  "recovery": "Fix the fields listed in errors[]."
}
  • code is stable and kebab-case; type ends with it. retryable tells clients whether the same request may succeed later.
  • Validation errors are 400 validation-error (never FastAPI's 422). errors[].pointer is a JSON Pointer into the body (/slug) or #/query/<name>, #/header/<name>, #/path/<name> for other inputs. Input values are never echoed. Unknown body fields and unknown query parameters are errors.
  • 422 is reserved for well-formed requests that cannot be processed (idempotency-key-reuse, invalid-expiry).

Request id, version and tracing

  • X-Request-ID: the client's value is kept when it matches ^[A-Za-z0-9_-]{8,64}$; otherwise the server generates req_…. It is in every response, problem, log line and audit event.
  • DX-Version: 2026-11-01: every response carries it; a request may send it, and an unsupported value is 400 unsupported-version. Major versions live in the path (/v1); within /v1 only additive changes are allowed.
  • traceparent (W3C): the trace id is propagated to the logs, the audit log and, when OTEL_EXPORTER_OTLP_ENDPOINT is set, to OpenTelemetry spans.
curl -s "$DX/v1/projects" -H "Authorization: Bearer $DX_API_KEY" \
  -H "DX-Version: 2026-11-01" -H "X-Request-ID: my-job-0042" \
  -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" -i

Collections: keyset pagination

curl -s "$DX/v1/audit-events?limit=100&action=project.created" \
  -H "Authorization: Bearer $DX_API_KEY"
# {"data": [...], "next_cursor": "eyJr…", "has_more": true}
curl -s "$DX/v1/audit-events?limit=100&action=project.created&cursor=eyJr…" \
  -H "Authorization: Bearer $DX_API_KEY"
  • limit defaults to 50; values above 200 are clipped and X-Limit-Applied reports the applied value. There is no offset and no total count.
  • Order is (created_at, id). The cursor is opaque and signed; it is only valid with the same filters. A tampered or foreign cursor is 400 invalid-cursor. Rows inserted while a client walks a collection never make it repeat or skip an item.

Idempotency

Every POST with effects accepts Idempotency-Key (1–255 printable characters; 24 h; scoped to the key's project):

curl -s -X POST "$DX/v1/projects" -H "Authorization: Bearer $DX_API_KEY" \
  -H "Idempotency-Key: 7c1d0f7e-create-invoices" -H "Content-Type: application/json" \
  -d '{"slug": "invoices", "name": "Invoices"}'
  • A retry with the same key and the same request (method, path, query and canonical body) replays the stored response with Idempotent-Replayed: true.
  • The same key with a different request: 422 idempotency-key-reuse. While the original is still running, a duplicate waits briefly and then gets 409 idempotency-key-in-flight (retryable).
  • A request that fails releases its key, so the client can retry. After 24 h the key is free. This follows the IETF Idempotency-Key draft (422/409) rather than a single 409.

Concurrency: ETag and If-Match

Editable resources return a strong ETag ("v<row_version>").

curl -s -i "$DX/v1/projects/prj_…" -H "Authorization: Bearer $DX_API_KEY"   # ETag: "v3"
curl -s -X PATCH "$DX/v1/projects/prj_…" -H "Authorization: Bearer $DX_API_KEY" \
  -H 'If-Match: "v3"' -H "Content-Type: application/json" -d '{"name": "Invoices EU"}'
  • PUT, PATCH and DELETE of editable resources require If-Match: missing → 428 precondition-required; stale → 412 precondition-failed.
  • GET with a matching If-None-Match → 304.

Rate limits and quotas

  • Per project and class: submit 120/min, read 600/min, review 300/min (plan defaults, api/plans.toml, overridable per project in project_limits). Responses carry RateLimit-Policy: "read";q=600;w=60 and RateLimit: "read";r=599;t=1; exceeding a class is 429 rate-limited with Retry-After.
  • Quotas (max_runs_per_day, max_pages_per_month, …; NULL means unlimited) are consumed inside the request transaction; exceeding one is 429 quota-exceeded with limit, used, resets_at and Retry-After.
  • The rate state lives in each replica's memory: with N replicas the effective limit is N times the configured one. Quotas are exact (database).

Audit

Every mutation writes one audit_log event (actor, action, target, outcome, request_id, trace_id, hashes of the before/after snapshots) in the same transaction as the change: if the change rolls back, so does the event. dx_app can only insert and read audit_log (a trigger rejects UPDATE and DELETE for every role). Denied requests with a known tenant are recorded as auth.denied (at most one per credential and reason per minute).

curl -s "$DX/v1/audit-events?actor=key_…&since=2026-11-01T00:00:00Z" \
  -H "Authorization: Bearer $DX_API_KEY"

Declaring a new endpoint

  1. Create src/extractit/api/routers/<resource>.py with router = dx_router(tags=[...]) and add the module name to ROUTER_MODULES in api/routers/__init__.py (one line).
  2. Declare each operation with an explicit snake_case operation_id, a summary, a description and a response_model, and require a permission:
@router.post(
    "/v1/widgets",
    operation_id="create_widget",
    summary="Create a widget",
    description="…",
    status_code=201,
    response_model=WidgetOut,
)
def create_widget(
    body: WidgetCreate,
    session: SessionDep,
    audit: AuditDep,
    principal: Annotated[Principal, require(Permission.WORKFLOW_WRITE, audit="widget.created")],
) -> WidgetOut:
    ...
    audit.record("widget", out.id, after=snapshot(widget))
    return out
  • require(permission, audit=..., rate_class=...) is mandatory on every route except PUBLIC_ROUTES (/healthz, /readyz, /openapi.json); non-GET routes must name an audit action (resource.verb). The route class refuses to build otherwise and tests/api/test_api_architecture.py checks the inventory.
  • The rate class defaults to read for GET and submit otherwise (rate_class="review" for review actions).
  • Use SessionDep (the request's tenant session: never open sessions in handlers), PageDep + paginate(...) for collections, ETagged for editable resources, QuotaDep to consume quotas, and raise ApiError("code", detail=...) for errors.
  • Organization-level resources use require(..., tenant_wide=True).
  • Tag operations that the MCP server (T47) should expose with mcp: they need a summary, a description and an operationId of at most 64 characters (Spectral enforces it).
  • Regenerate the snapshot and add a changelog entry:
uv run python -m extractit.api.openapi --out openapi/openapi.json
$EDITOR docs/changelog/api.md

Compatibility and deprecation

  • /v1 only changes additively. CI (api-contract) regenerates the snapshot (no diff allowed), runs Spectral (0 errors), runs oasdiff breaking against main (fails on removed endpoints or response fields, new required inputs…) and requires an entry in docs/changelog/api.md whenever the snapshot changes.
  • To deprecate an operation, add **deprecated(on="2027-01-15", sunset="2027-07-15") to its route. Responses then carry Deprecation: @<epoch> (RFC 9745) and Sunset: <HTTP-date> (RFC 8594); the sunset must be at least 180 days after the deprecation, checked when the route is built and by python -m extractit.api.openapi --check.

Observability

  • Logs are JSON (structlog) with request_id, trace_id, tenant_id and project_id. Values of sensitive keys (authorization, api_key, secret, token, password, …) and anything shaped like an API key are redacted; request and response bodies are never logged.
  • Traces: set OTEL_EXPORTER_OTLP_ENDPOINT to export OTLP/HTTP spans (server spans with dx.request_id, database spans without parameters). Without it nothing is created.