Skip to content

Observability

OpenTelemetry is the only contract. API and workers export OTLP/HTTP to the Collector (OTEL_EXPORTER_OTLP_ENDPOINT); without that variable they export nothing. Customer-facing data (run timeline, costs) comes from ExtractIt's own tables, never from the tracing backend.

The Collector

deploy/observability/otelcol.yaml (Compose) and the chart's ConfigMap share the same processors block (deploy/helm/extractit/files/otelcol-processors.yaml, kept identical by a test):

  • Content never leaves the Collector. attributes/drop-content deletes gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.definitions, the legacy gen_ai.prompt*/gen_ai.completion* keys, db.query.parameters and dx.content*, on spans, logs and metric data points; transform/drop-content does the same inside span events.
  • Zero data retention. For a tenant marked dx.zdr=true (resource or span) only identifiers (tenant_id, project_id, run_id, node_id, dx.* ids), timing, status and gen_ai usage attributes survive; span-event attributes are dropped and log bodies replaced by [redacted: zdr] — even when content capture was turned on in the process.
  • Tail sampling keeps every error and every trace slower than 30 s, and DX_OTEL_SAMPLE_PERCENT (default 10 %) of the rest.
  • Span metrics (dx_calls_total, dx_duration_* by route and status) are computed from every API server span before sampling: the input of the availability SLO.
  • Queue metrics (dx_queue_queued_runs, dx_queue_running_runs, dx_queue_oldest_age_seconds) come from the ops.queue_backlog view (receiver sqlquery, role dx_dbos).

If the Collector is down, API and runs continue: the SDK exporters retry briefly and drop.

Proof: tests/ops/test_ops_observability.py runs the pinned Collector image, sends traces, metrics and logs carrying the canary CANARY-7f3a91 in content attributes and for a ZDR tenant, and asserts 0 occurrences in every exported file and in what reaches a stand-in Langfuse OTLP endpoint, while ids and gen_ai usage attributes are kept.

Exporters

observability.mode Chart Compose
otlp Collector → observability.otlp.endpoint (any OTLP backend) and/or Langfuse —
bundle Collector + Jaeger compose.observability.yaml: Collector, Prometheus (SLO rules), Jaeger
none no Collector; processes export nothing default

Langfuse (MIT) needs ClickHouse, Valkey and S3, so it is deployed separately with its own chart or Compose; ExtractIt's Collector only needs its OTLP endpoint (langfuse.otlpEndpoint, e.g. https://langfuse.example.com/api/public/otel) and a Secret with LANGFUSE_AUTH (Basic base64(public_key:secret_key)). Compose: compose.langfuse.yaml.

Grafana, Loki and Tempo are AGPL-3.0: they are not bundled or modified. Use them as external backends if you like: Grafana on Prometheus/Jaeger, or Tempo/Loki as the otlp endpoint.

SLOs and alerts

deploy/observability/rules/slo.yaml (Prometheus rules), tested with promtool test rules deploy/observability/tests/slo_rules_test.yaml:

Alert Condition
ApiAvailabilityBurnRateFast (page) 5xx ratio > 14.4 × 0.1 % over 1 h and 5 min (SLO 99.9 % / 30 days)
ApiAvailabilityBurnRateSlow (ticket) 5xx ratio > 6 × 0.1 % over 6 h and 30 min
QueueBacklog (page) p95 over 10 min of the oldest queued run's age > 300 s for 10 min

The tests show that 2 % errors for an hour pages, 0.05 % does not, 1 % for 7 h opens a ticket and a 400 s queue age fires QueueBacklog.

Status page

GET /v1/status (public, cached 5 s, numbers and fixed reason codes only):

{"status": "up", "checked_at": "…",
 "components": [{"name": "database", "status": "up", "detail": {}},
                {"name": "queue", "status": "up", "detail": {"queued": 3, "running": 2, "oldest_queued_age_seconds": 4.1}},
                {"name": "storage", "status": "up", "detail": {}},
                {"name": "workers", "status": "up", "detail": {"executors": 2, "last_activity_age_seconds": 3.0}}]}

down when PostgreSQL does not answer within 5 s, the object store fails, the oldest queued run waits ≥ 30 min, or no worker touched a workflow for 10 min (the runtime reconciles every 30 s); degraded from 5 min queue age or 2 min worker silence. Gatus (deploy/status/gatus.yaml, or the chart's status.gatus.enabled) probes /healthz, /readyz and /v1/status; run one instance outside the deployment it watches.