Commit 8419569
committed
feat(ingest): add
Drives the dlthubworker ingest API (/v1/ingest/*) through the gateway via
a raw-HTTP client (routes aren't in the SDK yet): sql/rest/filesystem/
iceberg onboarding, restricted SQL queries, translate, connector catalog,
status, drain, schema, preview, and parquet download.
Reconciled against the stable post-gateway API (2026-07-07 rollout):
- enqueue (POST /sources|/queries) sends the durable hd_ API key as the
bearer — the server rejects JWTs there because the drain job outlives
them; without an API key the CLI fails fast with a --api-key hint
- verified live: the CLI's user-scoped session JWT only reaches
/connectors — workspace-scoped endpoints 403 (the worker derives the
workspace from the JWT's workspace:* scope and ignores X-Workspace-Id
on the JWT route), so those 403s get an --api-key hint too
- no `ingest sources` command: GET /ingest/sources never shipped
- 2.5s status polling, drain re-kick after 30s pending (control-store
read lag), brief 404 retry on `ingest query` right after onboarding,
and a database-scoped-token hint when a load fails with Forbiddenhotdata ingest command group1 parent 03dc51b commit 8419569
6 files changed
Lines changed: 1634 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
132 | 133 | | |
133 | 134 | | |
134 | 135 | | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
135 | 151 | | |
136 | 152 | | |
137 | 153 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
0 commit comments