Skip to content

Commit 8419569

Browse files
committed
feat(ingest): add hotdata ingest command group
Drives the dlthubworker ingest API (/v1/ingest/*) through the gateway via a raw-HTTP client (routes aren't in the SDK yet): sql/rest/filesystem/ iceberg onboarding, restricted SQL queries, translate, connector catalog, status, drain, schema, preview, and parquet download. Reconciled against the stable post-gateway API (2026-07-07 rollout): - enqueue (POST /sources|/queries) sends the durable hd_ API key as the bearer — the server rejects JWTs there because the drain job outlives them; without an API key the CLI fails fast with a --api-key hint - verified live: the CLI's user-scoped session JWT only reaches /connectors — workspace-scoped endpoints 403 (the worker derives the workspace from the JWT's workspace:* scope and ignores X-Workspace-Id on the JWT route), so those 403s get an --api-key hint too - no `ingest sources` command: GET /ingest/sources never shipped - 2.5s status polling, drain re-kick after 30s pending (control-store read lag), brief 404 retry on `ingest query` right after onboarding, and a database-scoped-token hint when a load fails with Forbidden
1 parent 03dc51b commit 8419569

6 files changed

Lines changed: 1634 additions & 0 deletions

File tree

src/cli.rs

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ use crate::commands::context::ContextCommands;
44
use crate::commands::databases::DatabasesCommands;
55
use crate::commands::embedding_providers::EmbeddingProvidersCommands;
66
use crate::commands::indexes::IndexesCommands;
7+
use crate::commands::ingest::IngestCommands;
78
use crate::commands::jobs::JobsCommands;
89
use crate::commands::queries::QueriesCommands;
910
use crate::commands::query::QueryCommands;
@@ -132,6 +133,21 @@ pub enum Commands {
132133
command: Option<JobsCommands>,
133134
},
134135

136+
/// Ingest data from external connectors (databases, REST APIs, files, Iceberg)
137+
/// into a managed database via the hotdata ingest service
138+
Ingest {
139+
/// Workspace ID (defaults to first workspace from login)
140+
#[arg(long, short = 'w', global = true)]
141+
workspace_id: Option<String>,
142+
143+
/// Output format
144+
#[arg(long = "output", short = 'o', default_value = "table", value_parser = ["table", "json", "yaml"], global = true)]
145+
output: String,
146+
147+
#[command(subcommand)]
148+
command: IngestCommands,
149+
},
150+
135151
/// Manage indexes on a table
136152
Indexes {
137153
/// Workspace ID (defaults to first workspace from login)

src/client.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
pub mod credentials;
22
pub mod database_session;
3+
pub mod ingest;
34
pub mod jwt;
45
pub mod raw_http;
56
pub mod sdk;

0 commit comments

Comments
 (0)