Storage connectors
Connectors move files in and out of ExtractIt without a person uploading anything:
- Ingestion (ING-07, ING-08): a connector watches a folder or mailbox and starts one run of a workflow (by alias) per new file — continuously, or once for a bulk import from Drive or OneDrive.
- Export (WF-17): the
exportnode sends the result of a run to S3, SFTP, Drive, OneDrive / SharePoint, a local folder or an ERP webhook.
Both use the same resource, the connection: the credentials (sealed per tenant) plus the
configuration of one remote. A connector binds a connection and a path to a workflow reference
name@alias. The page Settings → Connectors manages both.
Connections
| Kind | Reaches | Direction | Credentials |
|---|---|---|---|
drive |
Google Drive (shared drives too) | source and destination | OAuth (PKCE) or a token pasted from rclone authorize |
onedrive |
OneDrive and SharePoint libraries | source and destination | OAuth (PKCE) or a pasted token |
sftp |
an SFTP server | source and destination | password or private key; the host key is mandatory |
s3 |
Amazon S3, MinIO, Wasabi, Ceph… | source and destination | access key id + secret |
local |
a folder on the worker | source and destination | none (inside DX_WATCH_ROOTS) |
imap |
a mailbox over IMAP | source | password |
graph_mail |
a Microsoft 365 mailbox | source | application credentials (Mail.Read) |
webhook |
your ERP | destination | signing secret (whsec_…) |
GET /v1/connection-kinds returns, per kind, the JSON Schema of its configuration, its secret
fields and its OAuth provider; the settings page draws its forms from it.
Secrets are write-only. They are split off the configuration, sealed with the tenant data key
(AES-256-GCM, AAD organization | project | connection) and never appear in a response, a log, a
trace or a database dump. A response lists secret_fields_set and the page shows those fields as
saved; a PATCH that omits a secret keeps it. Reading needs project:read; changing a
connection, running a connector and authorizing need project:manage. allow_private_endpoint
(SFTP, S3 endpoints, IMAP, webhooks that resolve to a private network) can only be set by an
organization administrator.
access is read or read_write. Only read_write connections can be export destinations: a
read-only one used as a destination fails with CONNECTION_READ_ONLY (422).
rclone
Storage remotes are reached through rclone (MIT), an external binary
pinned in src/extractit/connectors/pins.toml (version and the sha256 of the official release)
and installed by the rclone stage of the root Dockerfile. scripts/check_rclone_pins.py fails
when the Dockerfile and pins.toml differ. Each worker process starts one rclone rcd:
- it listens on
127.0.0.1only, on a free port, with a random user and password per process (given throughRCLONE_RC_USER/RCLONE_RC_PASS, never the command line);--rc-no-authand--rc-serveare never used, andcore/commandandconfig/*are never called — the rc API is equivalent to a shell for the process user; - the configuration file is empty and ephemeral: the remotes travel inline as JSON objects in
fs/srcFs/dstFs({"type": "s3", …, "_root": "bucket/prefix"}), so norclone.confwith tokens ever exists.
For Drive and OneDrive the application refreshes the OAuth token (under a row lock, persisting the
rotated refresh token) and hands rclone only an access token with its expiry. A Google Doc,
Sheet or Slide is listed with a .pdf name and exported to PDF (export_formats=pdf).
H6: inline remotes with a token
The design depends on operations/list (with recurse, showHash and _filter.MaxAge),
operations/copyfile, operations/stat and core/obscure accepting inline remotes that carry a
token. This is verified with the pinned version in the connectors integration job
(pytest -m integration_connectors; it ran against rclone v1.76.0 with local, SFTP through
rclone serve sftp and S3 through a moto server: list, copyfile, stat and core/obscure
accept inline remotes. Drive and OneDrive remotes are only exercised against the fake rc
server). Two findings from that run: the listing Path is relative to the remote's
_root, so the connector path always goes in _root and remote stays empty; and rclone
1.76 cannot upload to a plain-HTTP S3 endpoint with a signed payload (request stream is
not seekable), so fs_for sets use_unsigned_payload=true when the endpoint starts with
http:// (TLS endpoints keep signed payloads). If a remote type refuses it, the plan B is a
configuration file with mode 0600 generated in tmpfs per call. The validation against real
Google Drive and OneDrive/SharePoint accounts (listing, download and upload) is manual and an
external dependency.
The local kind does not need rclone: the worker lists and copies the folder natively. The path
is resolved with realpath and must be inside DX_WATCH_ROOTS (a list of folders separated by :;
empty means no local connection is allowed), symbolic links are skipped, and an upload is written to
a temporary name and renamed so a watcher on the same folder never sees half a file.
OAuth applications
Each deployment (or each customer, when self-hosting) uses its own OAuth application. Put the client
id (and secret) in the connection, or set DX_OAUTH_GOOGLE_CLIENT_ID/_CLIENT_SECRET and
DX_OAUTH_MICROSOFT_CLIENT_ID/_CLIENT_SECRET for the instance, plus DX_PUBLIC_URL (the
redirect URI is $DX_PUBLIC_URL/v1/connections/oauth/callback).
Google. In Google Cloud Console create an OAuth client of type Web application, add the
redirect URI above, enable the Drive API and add the scope drive.readonly (ingestion) or drive
(export). Restricted-scope verification of the application for a hosted SaaS is outside this
ticket; a self-hosted customer uses its own application in Testing or Internal mode.
Microsoft. In Entra ID register an application (Web), add the redirect URI, create a client
secret and grant the delegated permissions offline_access, Files.Read.All and Sites.Read.All
(Files.ReadWrite.All and Sites.ReadWrite.All for export). Prefer Sites.Selected with an
explicit grant on the one site when the tenant allows it. An administrator may need to consent.
For SharePoint choose drive_type=documentLibrary and give the site_url and library: the
drive_id is resolved with Microsoft Graph once the connection is authorized.
Minimum permissions for mailboxes: Microsoft Graph application permission Mail.Read, limited to
the mailbox with an application access policy.
The authorization flow: POST /v1/connections/oauth/{google|microsoft}:start stores the state and a
PKCE (S256) verifier (sealed, 10 minutes) and returns the authorization_url; the page opens it in a
popup, the provider redirects to GET /v1/connections/oauth/callback, which validates the state
(single use), exchanges the code and marks the connection active. The page polls
GET /v1/connections/{id} until it is active or error; a blocked popup is reported, and pasting
the token from rclone authorize is the alternative without a browser.
Connectors
POST /v1/connectors takes connection_id, path, workflow_ref (name@alias; 422
ALIAS_NOT_FOUND if it does not resolve to a published version), poll_interval_s (default 60,
minimum DX_CONNECTOR_MIN_POLL_S, 10), duplicate_policy, settle_s, mode (continuous or
once) and ingest_existing.
A continuous connector is a DBOS scheduled task connector:<id> (cron with seconds when the
interval is below a minute, automatic_backfill=false). Pausing, resuming and deleting the
connector pause, resume and delete the schedule. Each tick:
- lists what is new since the checkpoint and drops what was already persisted;
- keeps only stable files — the same size as at the previous poll and an age of at least
settle_s(two polls at least); - starts one child workflow per element with the deterministic id
ing-<connector>-<hash>on a queue of concurrency 4, so a re-listing or a restart cannot duplicate; - each child downloads to a 0700 spool (deleted in a
finally), computes the sha256, applies the duplicate policy, creates the packet and theconnector_itemsrow in one transaction and starts the run with the idempotency keyconn:<connector>:<hash of remote_id and etag>. A crash between the packet and the run leaves a row without a run: the retry submits only the run; - a final step advances the checkpoint with the elements that were persisted only.
A connector runs one tick at a time: a tick takes a lease (connectors.tick_lease_until,
DX_CONNECTOR_TICK_LEASE_S, default 120 s) that finish_tick releases, so a schedule that fires
every second does not pile up ticks while a long one is still ingesting (the children of one tick
are awaited together, not one polling interval each). If the worker dies the lease expires and
the next tick takes over. An explicit :run ignores it.
Checkpoints by type: files use a modification-time watermark with a 10-minute margin (idempotency
absorbs the overlap); IMAP uses {uidvalidity, last_uid} (a changed UIDVALIDITY restarts the cursor,
and the Message-ID based remote_id keeps it from duplicating); Graph stores the deltaLink. The
alias is resolved when each element is ingested: after moving prod, new elements use the new
version.
Duplicate policy (same sha256 again, other name): reuse links the run that already processed it,
reject creates nothing, reprocess runs it anew. An element over DX_MAX_FILE_BYTES is
skipped_too_large, an unsupported type skipped_unsupported, and failed ones are retried on later
ticks up to 5 times. A connector whose poll fails applies an exponential backoff from 1 s to 15 min,
shows state=error with last_error (no secrets) and does not affect the others. Remote names are
sanitized: no separators, .. or control characters.
mode=once is the one-off import from Drive or OneDrive: POST /v1/connectors/{id}:run
ingests what exists (at most 200 elements per call; truncated says there are more) and later runs
bring only the changes. :run also works on continuous connectors and answers with the summary
(new, duplicates, failed, skipped).
Size
rclone lists the whole folder at each poll (O(n)); the listing is narrowed with a modification-age filter and the minimum interval is 10 s. Keep a connector under about 50 000 files.
Mailboxes
IMAP (imap-tools) polls one folder without marking messages as read unless mark_seen is on. Each
message becomes one packet with the original .eml (the conversion of T16 turns it into body and
attachments) and the metadata {subject, from, message_id, received_at}. With require_attachments
(default) a message without a supported attachment is skipped_unsupported. Microsoft 365 uses the
messages/delta query of Graph (the deltaLink is the checkpoint) and downloads the MIME with
/messages/{id}/$value. Against GreenMail, which rejects UID SEARCH UID n:*, the source falls
back to SEARCH ALL filtered by UID, and the message is fetched with UID FETCH … BODY.PEEK[]
(no search, nothing marked as read).
The export node
export accepts Extraction, Verdicts and File, has no outputs, is class io and is not cached.
{
"destination": "facturas-s3",
"path": "{prefix}/{workflow_id}/{packet_id}/{run_id}.json",
"format": "json",
"mapping": {
"factura": "${fields.invoice_number}",
"importe": "${fields.total}",
"lineas": "${fields.line_items[*].amount}"
},
"exclude": ["debug"],
"on_failure": "fail_run"
}
destinationis the name (orcon_…id) of a connection. Placeholders are${fields.path}(the extraction'sdata),${fields.rows[*].col},${document.name|sha256|pages},${run.id}and${segment.id}; there is no Jinja and no expression evaluation. A template that is exactly one placeholder keeps the type.includeandexcludetake dotted paths.format: "csv"withrows_from: "fields.line_items"writes a header and one row per element (${row.amount}reads the element); a File input is uploaded as it is, with its name and sha256.pathaccepts{prefix},{workflow_id},{packet_id},{run_id},{node_id},{scope_key},{date},{name}and{ext}, and every segment is sanitized.- Publishing validates the mapping against the schema of the upstream extract nodes, the path
keys and the destination (it must exist and be
read_write): every violation comes back at once asEXPORT_MAPPING_INVALID,CONNECTION_NOT_FOUNDorCONNECTION_READ_ONLY. The destination check starts once the project has at least one connection. - Idempotency.
idempotency_key = sha256(run_id, node_id, scope_key, connection_id, mapping_hash, path), stored inexport_deliveries. Anexportedrow answersalready_exportedwithout calling the destination; astartedrow (a crash in between) repeats the write on the same deterministic path. - Retries. Up to 6 attempts, exponential backoff from 2 s with jitter,
Retry-Afterhonoured (at most 120 s), no retry of a 4xx except 408 and 429. When they are exhausted the node fails withEXPORT_FAILEDandretryable=true—POST /v1/runs/{id}:resumeruns only this node — or, withon_failure: "continue_with_warning", succeeds with a warning. - Webhook.
POSTof the JSON (at most 5 MiB) withIdempotency-Keyand the Standard Webhooks headerswebhook-id,webhook-timestampandwebhook-signature(secretwhsec_…), through the SSRF-safe route of the import service: the validated IP is pinned in the URL, there is no proxy and redirects are never followed; loopback, link-local and private addresses (and DNS rebinding to them) are refused unlessallow_private_endpointis set.
Operations
| Variable | Default | Meaning |
|---|---|---|
DX_RCLONE_BIN |
rclone on PATH |
the pinned binary |
DX_WATCH_ROOTS |
empty | folders a local connection may use |
DX_CONNECTOR_SPOOL |
$TMPDIR/dx-spool |
spool root (0700) |
DX_CONNECTOR_MIN_POLL_S |
10 | smallest accepted poll_interval_s |
DX_CONNECTOR_TICK_LEASE_S |
120 | how long a running tick holds its connector |
DX_MAX_FILE_BYTES |
plan limit (25 MB) | bigger elements are skipped_too_large |
DX_PUBLIC_URL |
http://127.0.0.1:8000 |
base of the OAuth redirect URI |
DX_OAUTH_<GOOGLE\|MICROSOFT>_CLIENT_ID/_SECRET |
— | the instance's OAuth application |
DX_OAUTH_<…>_AUTH_URL/_TOKEN_URL |
the provider's | point at a simulated IdP (tests) |
DX_GRAPH_URL |
https://graph.microsoft.com/v1.0 |
Microsoft Graph base |
The worker loads the connector workflows through the extractit.run_handlers entry points
connectors and export_node. Tests of the integrations run with pytest -m integration_connectors
(rclone, GreenMail and a moto server; the moto server needs moto[server], i.e. flask); everything else uses the fake rc server in
tests/fixtures/connectors/fake_rc.py.