diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 57a89d783..c81fce637 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -34,8 +34,10 @@ This page tracks significant updates to the QuestDB documentation. ### Updated +- [Storage policy](/docs/concepts/storage-policy/) - Removed the `DROP NATIVE` stage; `TO PARQUET` now removes native files and serves reads from the Parquet file - [SQL extensions and compatibility](/docs/concepts/deep-dive/sql-extensions/) - Rewritten as the canonical SQL dialect reference covering time-series clauses and joins, query syntax conveniences, storage extensions, PIVOT vs SQL Server/PostgreSQL, JSON and UNNEST, and a full inventory of standard SQL features not supported with QuestDB equivalents. Sidebar entry moved to the top of Query & SQL Reference - [systemd service example](/docs/deployment/systemd/) - Corrected JVM argument ordering, switched to `UseParallelGC`, set `-Dcontainerized=false`, and added required setup paths +- [Materialized views](/docs/query/sql/create-mat-view/) - Removed storage policy support for materialized views; use `TTL` for retention instead - [Aggregation functions](/docs/query/functions/aggregation/) - Added demo tags and updated examples with runnable queries - [LATEST ON](/docs/query/sql/latest-on/) - Added demo tags to examples - [JOIN](/docs/query/sql/join/) - Updated examples to use demo data diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 4010bf03e..a0722c58b 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -336,16 +336,6 @@ CREATE MATERIALIZED VIEW trades_hourly AS ( The view's TTL is independent of the base table's TTL. -:::note - -In QuestDB Enterprise, TTL is superseded by -[storage policies](/docs/concepts/storage-policy/). Use -[`STORAGE POLICY(...)`](/docs/query/sql/alter-mat-view-set-storage-policy/) on -a materialized view instead of `TTL` for graduated lifecycle management -(convert to Parquet, then drop). - -::: - ### Initial refresh When created, materialized views start an **asynchronous full refresh**: @@ -602,9 +592,6 @@ the replica's view was not fully up-to-date. Sets the time limit for incremental refresh on a materialized view - [`ALTER MATERIALIZED VIEW SET TTL`](/docs/query/sql/alter-mat-view-set-ttl/): Sets the time-to-live (TTL) period on a materialized view - - [`ALTER MATERIALIZED VIEW SET STORAGE POLICY`](/docs/query/sql/alter-mat-view-set-storage-policy/): - Attaches a [storage policy](/docs/concepts/storage-policy/) to a - materialized view (QuestDB Enterprise) - **Configuration** - [Materialized views configs](/docs/configuration/materialized-views/): diff --git a/documentation/concepts/storage-policy.md b/documentation/concepts/storage-policy.md index aaa27a490..a0ecc70cb 100644 --- a/documentation/concepts/storage-policy.md +++ b/documentation/concepts/storage-policy.md @@ -11,19 +11,19 @@ Storage policies are available in **QuestDB Enterprise** only. ::: A storage policy automates the lifecycle of table partitions. It defines when -partitions are converted to Parquet, when native data is removed, and when local -copies are dropped. This replaces the need for manual partition management or -external scheduling. +partitions are converted to Parquet and when local copies are dropped. +Converting a partition to Parquet removes its native files and serves reads directly from the Parquet file. This replaces the need for manual partition management or external scheduling. :::info Storage policies currently operate **locally only**. Parquet files are not -automatically uploaded to object storage, and the `DROP REMOTE` clause is -reserved syntax — it is rejected at SQL parse time with -`'DROP REMOTE' is not supported yet`. Accordingly, the `drop_remote` -column in the [`storage_policies`](/docs/query/functions/meta/#storage_policies) -view is always blank in the current release; it is kept for forward -compatibility. Object storage integration will be added in a future release. +automatically uploaded to object storage, so the `TO REMOTE` and `DROP REMOTE` +clauses are reserved syntax — they are rejected at SQL parse time with +`'TO REMOTE' is not supported yet` and `'DROP REMOTE' is not supported yet`. +Accordingly, the `to_remote` and `drop_remote` columns in the +[`storage_policies`](/docs/query/functions/meta/#storage_policies) view are +always blank in the current release; they are kept for forward compatibility. +Object storage integration will be added in a future release. ::: @@ -40,12 +40,12 @@ Storage policies require: A storage policy consists of up to four TTL settings. Each setting controls a stage in the partition lifecycle: -| Setting | Description | -|---------|-------------| -| `TO PARQUET` | Convert the partition from native binary format to Parquet | -| `DROP NATIVE` | Remove native binary files, keeping only the local Parquet copy | -| `DROP LOCAL` | Remove all local data (both native and Parquet) | -| `DROP REMOTE` | _Reserved._ Will remove the Parquet file from object storage when remote upload is supported | +| Setting | Description | +| ------------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| `TO PARQUET` | Convert the partition from native binary format to Parquet. The native files are removed and reads are served from the Parquet file | +| `TO REMOTE` | _Reserved._ Will upload the Parquet file to object storage when remote upload is supported | +| `DROP LOCAL` | Remove all local data (native or Parquet) | +| `DROP REMOTE` | _Reserved._ Will remove the Parquet file from object storage when remote upload is supported | All settings are optional. Use only the ones relevant to your use case. All TTL values must be **positive**; `0` is rejected. @@ -56,12 +56,11 @@ As time passes, each partition progresses through the stages defined by the policy: ```text - TO PARQUET DROP NATIVE DROP LOCAL - [Native] ──────────┬──────────────────┬──────────────────┬─────── - │ │ │ - ▼ ▼ ▼ - Native + Parquet Parquet only Data removed - (local) (local) + TO PARQUET DROP LOCAL + [Native] ───────────────┬──────────────────────────┬─────── + │ │ + ▼ ▼ + Parquet only (local) Data removed ``` ### TTL evaluation @@ -75,10 +74,10 @@ eligible when: partition_end_time < reference_time - TTL ``` **This rule is applied independently for each stage's TTL.** A partition can -be eligible for `TO PARQUET` long before it is eligible for `DROP NATIVE`, -`DROP LOCAL`, or (one day) `DROP REMOTE`. Each stage uses its own `TTL` in the -formula above; the stages share only the reference time and the ordering -constraint `TO PARQUET <= DROP NATIVE <= DROP LOCAL <= DROP REMOTE`. +be eligible for `TO PARQUET` long before it is eligible for `DROP LOCAL` (or, +one day, the reserved `TO REMOTE` and `DROP REMOTE` stages). Each stage uses +its own `TTL` in the formula above; the stages share only the reference time +and the [ordering constraints](#ordering-constraint). The reference time is `min(wall_clock_time, latest_timestamp)` by default — the same formula used by TTL. The @@ -96,12 +95,12 @@ processes eligible partitions automatically. Storage policies replace [TTL](/docs/concepts/ttl/) in QuestDB Enterprise. If you are already familiar with TTL, this comparison is the fastest way in: -| | TTL | Storage Policy | -|---|-----|----------------| -| **Availability** | Open source | Enterprise only | -| **Action** | Drops partitions entirely | Graduated lifecycle (convert, then drop) | -| **Parquet conversion** | No | Yes (automatic local conversion) | -| **Granularity** | Single retention window | Up to four independent TTL stages | +| | TTL | Storage Policy | +| ---------------------- | ------------------------- | ---------------------------------------- | +| **Availability** | Open source | Enterprise only | +| **Action** | Drops partitions entirely | Graduated lifecycle (convert, then drop) | +| **Parquet conversion** | No | Yes (automatic local conversion) | +| **Granularity** | Single retention window | Up to four independent TTL stages | In QuestDB Enterprise, `CREATE TABLE ... TTL` and `ALTER TABLE SET TTL` are deprecated. Use storage policies instead: @@ -133,7 +132,7 @@ CREATE TABLE trades ( symbol SYMBOL, price DOUBLE ) TIMESTAMP(ts) PARTITION BY DAY - STORAGE POLICY(TO PARQUET 3d, DROP NATIVE 10d, DROP LOCAL 1M) + STORAGE POLICY(TO PARQUET 3d, DROP LOCAL 1M) WAL; ``` @@ -142,28 +141,12 @@ CREATE TABLE trades ( ```questdb-sql ALTER TABLE trades SET STORAGE POLICY( TO PARQUET 3 DAYS, - DROP NATIVE 10 DAYS, DROP LOCAL 1 MONTH ); ``` Only the specified settings are changed. Omitted settings remain unchanged. -### On materialized views - -```questdb-sql -CREATE MATERIALIZED VIEW hourly_trades AS ( - SELECT ts, symbol, sum(price) total - FROM trades - SAMPLE BY 1h -) PARTITION BY DAY - STORAGE POLICY(TO PARQUET 7d, DROP NATIVE 14d); -``` - -```questdb-sql -ALTER MATERIALIZED VIEW hourly_trades SET STORAGE POLICY(TO PARQUET 7d); -``` - For full syntax details, see [ALTER TABLE SET STORAGE POLICY](/docs/query/sql/alter-table-set-storage-policy/). @@ -172,24 +155,29 @@ For full syntax details, see Storage policy TTLs accept the same duration formats as [TTL](/docs/concepts/ttl/): -| Unit | Long form | Short form | -|------|-----------|------------| -| Hours | `1 HOUR` / `2 HOURS` | `1h` | -| Days | `1 DAY` / `3 DAYS` | `1d` / `3d` | -| Weeks | `1 WEEK` / `2 WEEKS` | `1W` / `2W` | +| Unit | Long form | Short form | +| ------ | ---------------------- | ----------- | +| Hours | `1 HOUR` / `2 HOURS` | `1h` | +| Days | `1 DAY` / `3 DAYS` | `1d` / `3d` | +| Weeks | `1 WEEK` / `2 WEEKS` | `1W` / `2W` | | Months | `1 MONTH` / `6 MONTHS` | `1M` / `6M` | -| Years | `1 YEAR` / `2 YEARS` | `1Y` / `2Y` | +| Years | `1 YEAR` / `2 YEARS` | `1Y` / `2Y` | ### Ordering constraint -TTL values must be in ascending order: +The stages form a partial order, not a single chain. A drop stage may not fire +before the write stage it depends on: ```text -TO PARQUET <= DROP NATIVE <= DROP LOCAL <= DROP REMOTE +TO PARQUET <= DROP LOCAL +TO REMOTE <= DROP LOCAL <= DROP REMOTE ``` -For example, you cannot drop native files before the Parquet conversion -completes. All TTL values must be positive — `0` is rejected. +`TO PARQUET` and `TO REMOTE` are **independent** — neither has to precede the +other. If `TO REMOTE` fires before `TO PARQUET`, both the native and Parquet +copies are written locally and reads continue to be served from the native +format until `TO PARQUET` removes the native files. All TTL values must be +positive — `0` is rejected. ## Disabling and enabling @@ -224,9 +212,9 @@ Query the `storage_policies` system view to see all active policies: SELECT * FROM storage_policies; ``` -| table_dir_name | to_parquet | drop_native | drop_local | drop_remote | status | last_updated | -|----------------|-----------|-------------|------------|-------------|--------|--------------| -| trades~12 | 72h | 240h | 1m | | A | 2025-01-15T10:30:00.000000Z | +| table_dir_name | to_parquet | to_remote | drop_local | drop_remote | status | last_updated | +| -------------- | ---------- | --------- | ---------- | ----------- | ------ | --------------------------- | +| trades~12 | 72h | | 1m | | A | 2025-01-15T10:30:00.000000Z | - TTL values are rendered in just two units: `h` for hours and `m` for **months**. Hour-, day-, and week-based durations are normalized to hours @@ -236,9 +224,10 @@ SELECT * FROM storage_policies; has no unit for minutes - Status `A` means active; `D` means disabled (see [Disabling and enabling](#disabling-and-enabling)) -- Unset stages appear blank. `drop_remote` is **always blank in the current - release** because `DROP REMOTE` is rejected at SQL parse time with - `'DROP REMOTE' is not supported yet`; the column is kept for forward +- Unset stages appear blank. `to_remote` and `drop_remote` are **always blank + in the current release** because `TO REMOTE` and `DROP REMOTE` are rejected + at SQL parse time with `'TO REMOTE' is not supported yet` and + `'DROP REMOTE' is not supported yet`; the columns are kept for forward compatibility For the full column reference and types, see @@ -264,25 +253,25 @@ Storage policy behavior can be tuned in `server.conf`. Time-based properties accept values with unit suffixes (e.g., `15m`, `30s`, `1h`) or raw microsecond values: -| Property | Default | Description | -|----------|---------|-------------| -| `storage.policy.check.interval` | `15m` (15 min) | How often QuestDB scans for partitions to process | -| `storage.policy.retry.interval` | `1m` (1 min) | Retry interval for failed tasks | -| `storage.policy.max.reschedule.count` | `20` | Maximum retries before abandoning a task | -| `storage.policy.writer.wait.timeout` | `30s` (30 sec) | Timeout for acquiring the table writer | -| `storage.policy.worker.count` | `2` | Number of storage policy worker threads (0 disables the feature) | -| `storage.policy.worker.affinity` | `-1` (no affinity) | CPU affinity for each worker thread (comma-separated list) | -| `storage.policy.worker.sleep.timeout` | `100ms` | Sleep duration when worker has no tasks | +| Property | Default | Description | +| ------------------------------------- | ------------------ | ---------------------------------------------------------------- | +| `storage.policy.check.interval` | `15m` (15 min) | How often QuestDB scans for partitions to process | +| `storage.policy.retry.interval` | `1m` (1 min) | Retry interval for failed tasks | +| `storage.policy.max.reschedule.count` | `20` | Maximum retries before abandoning a task | +| `storage.policy.writer.wait.timeout` | `30s` (30 sec) | Timeout for acquiring the table writer | +| `storage.policy.worker.count` | `2` | Number of storage policy worker threads (0 disables the feature) | +| `storage.policy.worker.affinity` | `-1` (no affinity) | CPU affinity for each worker thread (comma-separated list) | +| `storage.policy.worker.sleep.timeout` | `100ms` | Sleep duration when worker has no tasks | ## Permissions Storage policy operations require specific permissions in QuestDB Enterprise: -| Operation | Required permission | -|-----------|-------------------| -| `SET STORAGE POLICY` | `SET STORAGE POLICY` | -| `DROP STORAGE POLICY` | `REMOVE STORAGE POLICY` | -| `ENABLE STORAGE POLICY` | `ENABLE STORAGE POLICY` | +| Operation | Required permission | +| ------------------------ | ------------------------ | +| `SET STORAGE POLICY` | `SET STORAGE POLICY` | +| `DROP STORAGE POLICY` | `REMOVE STORAGE POLICY` | +| `ENABLE STORAGE POLICY` | `ENABLE STORAGE POLICY` | | `DISABLE STORAGE POLICY` | `DISABLE STORAGE POLICY` | Grant permissions using standard RBAC syntax: @@ -304,19 +293,19 @@ CREATE TABLE trades ( symbol SYMBOL, price DOUBLE ) TIMESTAMP(ts) PARTITION BY DAY - STORAGE POLICY(TO PARQUET 3d, DROP NATIVE 10d, DROP LOCAL 1M) + STORAGE POLICY(TO PARQUET 3d, DROP LOCAL 1M) WAL; ``` ```questdb-sql title="2. Verify via the system view" -SELECT table_dir_name, to_parquet, drop_native, drop_local, status +SELECT table_dir_name, to_parquet, drop_local, status FROM storage_policies WHERE table_dir_name LIKE 'trades%'; ``` -| table_dir_name | to_parquet | drop_native | drop_local | status | -|----------------|------------|-------------|------------|--------| -| trades~12 | 72h | 240h | 1m | A | +| table_dir_name | to_parquet | drop_local | status | +| -------------- | ---------- | ---------- | ------ | +| trades~12 | 72h | 1m | A | ```questdb-sql title="3. Modify one stage (others remain unchanged)" ALTER TABLE trades SET STORAGE POLICY(TO PARQUET 1d); @@ -326,13 +315,13 @@ ALTER TABLE trades SET STORAGE POLICY(TO PARQUET 1d); SHOW CREATE TABLE trades; ``` -```text +```questdb-sql CREATE TABLE 'trades' ( ts TIMESTAMP, symbol SYMBOL CAPACITY 256 CACHE, price DOUBLE ) timestamp(ts) PARTITION BY DAY -STORAGE POLICY(TO PARQUET 1 DAY, DROP NATIVE 10 DAYS, DROP LOCAL 1 MONTH) WAL; +STORAGE POLICY(TO PARQUET 1 DAY) WAL; ``` ```questdb-sql title="5. Temporarily suspend the policy (e.g. during a backfill)" @@ -348,16 +337,16 @@ ALTER TABLE trades DROP STORAGE POLICY; ## Guidelines -| Use case | Suggested policy | Rationale | -|----------|-----------------|-----------| -| Real-time metrics | `TO PARQUET 1d, DROP NATIVE 7d, DROP LOCAL 30d` | Keep recent data fast, drop old data automatically | -| Trading data | `TO PARQUET 7d, DROP NATIVE 30d` | Keep Parquet locally for long-term queries | -| IoT telemetry | `TO PARQUET 1d, DROP NATIVE 3d, DROP LOCAL 90d` | High volume, convert early to save disk; keep a brief native overlap for in-flight queries before dropping the native files | -| Aggregated views | `TO PARQUET 30d` | Low volume, keep locally in Parquet | +| Use case | Suggested policy | Rationale | +| ----------------- | ------------------------------- | ---------------------------------------------------------------- | +| Real-time metrics | `TO PARQUET 1d, DROP LOCAL 30d` | Keep recent data fast, drop old data automatically | +| Trading data | `TO PARQUET 7d` | Keep Parquet locally for long-term queries | +| IoT telemetry | `TO PARQUET 1d, DROP LOCAL 90d` | High volume, convert early to save disk before dropping the data | +| Aggregated views | `TO PARQUET 30d` | Low volume, keep locally in Parquet | **Tips:** -- Start with `TO PARQUET` and `DROP NATIVE` to reduce local disk usage while - keeping data queryable in Parquet format +- Start with `TO PARQUET` to reduce local disk usage while keeping data + queryable in Parquet format - Use `DROP LOCAL` with care as it permanently removes data from the local disk - TTL values should be significantly larger than the partition interval diff --git a/documentation/getting-started/enterprise-quick-start.md b/documentation/getting-started/enterprise-quick-start.md index 1e1f64483..59c3a5b4f 100644 --- a/documentation/getting-started/enterprise-quick-start.md +++ b/documentation/getting-started/enterprise-quick-start.md @@ -478,48 +478,48 @@ too — on a schedule you define. This supersedes plain TTL in Enterprise, where ### Migrating from TTL when upgrading from OSS -Tables and materialized views that were created in OSS keep their existing -`TTL` setting after you upgrade to Enterprise — no data is lost at upgrade -time. However, Enterprise rejects any **new** `TTL` changes (both -`CREATE ... TTL` and `ALTER ... SET TTL `) with: +Tables that were created in OSS keep their existing `TTL` setting after you +upgrade to Enterprise — no data is lost at upgrade time. However, Enterprise +rejects any **new** `TTL` changes on tables (both `CREATE TABLE ... TTL` and +`ALTER TABLE SET TTL `) with: ``` TTL settings are deprecated, please, create a storage policy instead ``` -To move a legacy table or materialized view from `TTL` to a storage policy: +Materialized views are not affected: they continue to use `TTL` for retention +in Enterprise. + +To move a legacy table from `TTL` to a storage policy: 1. **Clear the existing TTL** by setting it to `0`. This is the only `SET TTL` - value Enterprise accepts, and it is required before a storage policy can be - attached: + value Enterprise accepts on a table, and it is required before a storage + policy can be attached: ```questdb-sql title="Clear the legacy TTL" ALTER TABLE trades SET TTL 0; - -- or, for a materialized view: - ALTER MATERIALIZED VIEW trades_hourly SET TTL 0; ``` 2. **Attach a storage policy** that reproduces — and ideally extends — the retention the TTL used to provide. A policy lets you keep data in Parquet - after you would previously have dropped it, so `DROP LOCAL` (or - `DROP NATIVE` if you don't want Parquet at all) is the stage that replaces - the old TTL horizon: + after you would previously have dropped it, so `DROP LOCAL` is the stage + that replaces the old TTL horizon: ```questdb-sql title="Replace a 1-month TTL with an equivalent policy" ALTER TABLE trades SET STORAGE POLICY( TO PARQUET 3 DAYS, - DROP NATIVE 10 DAYS, DROP LOCAL 1 MONTH ); ``` If you want the policy to behave exactly like the old TTL (delete the - partition outright after the same interval), use a single-stage policy — - for example `STORAGE POLICY(DROP NATIVE 1 MONTH)` to match `TTL 1 MONTH`. + partition outright after the same interval, with no Parquet conversion), + use a single-stage policy — for example `STORAGE POLICY(DROP LOCAL 1 MONTH)` + to match `TTL 1 MONTH`. -Do this for every table and materialized view you want to keep managed -automatically. Tables without a storage policy retain their data indefinitely -once their legacy TTL has been cleared. +Do this for every table you want to keep managed automatically. Tables without +a storage policy retain their data indefinitely once their legacy TTL has been +cleared. ### Creating new tables with a storage policy @@ -531,7 +531,7 @@ CREATE TABLE trades ( symbol SYMBOL, price DOUBLE ) TIMESTAMP(ts) PARTITION BY DAY - STORAGE POLICY(TO PARQUET 3d, DROP NATIVE 10d, DROP LOCAL 1M) + STORAGE POLICY(TO PARQUET 3d, DROP LOCAL 1M) WAL; ``` @@ -540,7 +540,6 @@ Or attach a policy to an existing table: ```questdb-sql title="Web Console - Set a storage policy on an existing table" ALTER TABLE trades SET STORAGE POLICY( TO PARQUET 3 DAYS, - DROP NATIVE 10 DAYS, DROP LOCAL 1 MONTH ); ``` diff --git a/documentation/high-availability/overview.md b/documentation/high-availability/overview.md index c36f38bde..25b840f3a 100644 --- a/documentation/high-availability/overview.md +++ b/documentation/high-availability/overview.md @@ -82,9 +82,9 @@ replicated automatically and should be populated separately on each instance. ## Storage policies in a replicated cluster [Storage policy](/docs/concepts/storage-policy/) definitions are stored in -WAL-backed system tables, so the policy itself — the `TO PARQUET`, -`DROP NATIVE`, and `DROP LOCAL` TTLs and the active/disabled status — is -replicated to every instance through the same WAL pipeline as user data. +WAL-backed system tables, so the policy itself — the `TO PARQUET` and +`DROP LOCAL` TTLs and the active/disabled status — is replicated to every +instance through the same WAL pipeline as user data. Enforcement, however, runs **independently on each instance**. Parquet files are produced locally and are not replicated; each node's storage policy job diff --git a/documentation/operations/backup.md b/documentation/operations/backup.md index 012592d4e..5e982738b 100644 --- a/documentation/operations/backup.md +++ b/documentation/operations/backup.md @@ -335,13 +335,13 @@ To find your instance name, see [Backup instance name](#backup-instance-name). ### Interaction with storage policies [Storage policies](/docs/concepts/storage-policy/) operate locally — they -convert partitions to Parquet in place and then drop native (and eventually -local Parquet) files on a schedule. Backups capture whatever is on local disk -at the time the backup runs: +convert partitions to Parquet in place (removing the native files) and +eventually drop the local Parquet files on a schedule. Backups capture +whatever is on local disk at the time the backup runs: - Partitions still in native format are backed up as native files. -- Partitions that have been converted to Parquet (via the `TO PARQUET` stage, - after `DROP NATIVE` has fired) are backed up as Parquet files. +- Partitions that have been converted to Parquet (via the `TO PARQUET` stage) + are backed up as Parquet files. - Once `DROP LOCAL` fires and removes a partition from local disk, subsequent backups will no longer contain that partition — restoring an earlier backup is the only way to recover it. diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index e02f30f6b..3e8f30f1a 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -295,8 +295,8 @@ Storage policies — and the `storage_policies` view — are available in ::: `storage_policies` is a system view that lists every -[storage policy](/docs/concepts/storage-policy/) currently attached to a table -or materialized view. Query it like any other table: +[storage policy](/docs/concepts/storage-policy/) currently attached to a table. +Query it like any other table: ```questdb-sql SELECT * FROM storage_policies; @@ -306,9 +306,9 @@ SELECT * FROM storage_policies; | Column | Type | Description | |--------|------|-------------| -| `table_dir_name` | _STRING_ | Directory name of the table or materialized view the policy is attached to. Matches the `table_dir_name` column in [`tables()`](#tables) / [`materialized_views`](#materialized_views). | +| `table_dir_name` | _STRING_ | Directory name of the table the policy is attached to. Matches the `table_dir_name` column in [`tables()`](#tables). | | `to_parquet` | _STRING_ | TTL for the `TO PARQUET` stage (e.g. `72h`, `1m`). Blank when the stage is not configured. | -| `drop_native` | _STRING_ | TTL for the `DROP NATIVE` stage. Blank when the stage is not configured. | +| `to_remote` | _STRING_ | Reserved — always blank in the current release. The `TO REMOTE` clause is rejected at SQL parse time with `'TO REMOTE' is not supported yet`. The column is kept for forward compatibility. | | `drop_local` | _STRING_ | TTL for the `DROP LOCAL` stage. Blank when the stage is not configured. | | `drop_remote` | _STRING_ | Reserved — always blank in the current release. The `DROP REMOTE` clause is rejected at SQL parse time with `'DROP REMOTE' is not supported yet`. The column is kept for forward compatibility. | | `status` | _CHAR_ | Policy status. `A` = active (the policy is being enforced), `D` = disabled (via [`ALTER TABLE DISABLE STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/)). | @@ -329,14 +329,15 @@ SELECT * FROM storage_policies; SELECT * FROM storage_policies; ``` -| table_dir_name | to_parquet | drop_native | drop_local | drop_remote | status | last_updated | -|----------------|------------|-------------|------------|-------------|--------|--------------| -| trades~12 | 72h | 240h | 1m | | A | 2025-01-15T10:30:00.000000Z | -| metrics~18 | 168h | | | | D | 2025-01-14T09:15:42.000000Z | +| table_dir_name | to_parquet | to_remote | drop_local | drop_remote | status | last_updated | +|----------------|------------|-----------|------------|-------------|--------|--------------| +| trades~12 | 72h | | 1m | | A | 2025-01-15T10:30:00.000000Z | +| metrics~18 | 168h | | | | D | 2025-01-14T09:15:42.000000Z | -The first row is a policy with three active stages (3-day Parquet conversion, -10-day native drop, 1-month local drop) and is currently enforced. The second -row has only the `TO PARQUET` stage set and has been temporarily disabled. +The first row is a policy with two active stages (3-day Parquet conversion and +1-month local drop) and is currently enforced. The second row has only the +`TO PARQUET` stage set and has been temporarily disabled. The `to_remote` and +`drop_remote` columns are reserved and always blank in the current release. ## table_columns diff --git a/documentation/query/sql/alter-mat-view-set-storage-policy.md b/documentation/query/sql/alter-mat-view-set-storage-policy.md deleted file mode 100644 index 8628d23a1..000000000 --- a/documentation/query/sql/alter-mat-view-set-storage-policy.md +++ /dev/null @@ -1,162 +0,0 @@ ---- -title: ALTER MATERIALIZED VIEW SET STORAGE POLICY -sidebar_label: SET STORAGE POLICY -description: - ALTER MATERIALIZED VIEW SET STORAGE POLICY SQL keyword reference documentation. ---- - -Sets, modifies, enables, disables, or removes a storage policy on a materialized -view. - -:::note - -Storage policies are available in **QuestDB Enterprise** only. - -::: - -Refer to the [Storage Policy](/docs/concepts/storage-policy/) concept guide for -a full overview. - -## Syntax - -### Set or modify a storage policy - -```questdb-sql -ALTER MATERIALIZED VIEW view_name SET STORAGE POLICY( - [TO PARQUET ttl,] - [DROP NATIVE ttl,] - [DROP LOCAL ttl,] - [DROP REMOTE ttl] -); -``` - -Only the specified settings are changed. Omitted settings retain their current -values. - -### Enable or disable a storage policy - -```questdb-sql -ALTER MATERIALIZED VIEW view_name ENABLE STORAGE POLICY; -ALTER MATERIALIZED VIEW view_name DISABLE STORAGE POLICY; -``` - -Disabling a policy suspends processing without removing the policy definition. - -### Remove a storage policy - -```questdb-sql -ALTER MATERIALIZED VIEW view_name DROP STORAGE POLICY; -``` - -This permanently removes the storage policy from the materialized view. - -## Description - -A storage policy defines up to four TTL-based stages that control how partitions -transition from native format to Parquet and eventually get removed: - -| Setting | Effect | -|---------|--------| -| `TO PARQUET ` | Convert partition from native format to Parquet locally | -| `DROP NATIVE ` | Remove native binary files, keeping only the local Parquet copy | -| `DROP LOCAL ` | Remove all local copies of the partition | -| `DROP REMOTE ` | _Reserved._ Will remove the partition from object storage when remote upload is supported | - -:::info - -`DROP REMOTE` is reserved syntax. It is rejected at SQL parse time with -`'DROP REMOTE' is not supported yet`. Automatic upload of Parquet files to -object storage is not currently supported — storage policies operate locally -only. Because the clause cannot take effect, the `drop_remote` column in the -[`storage_policies`](/docs/query/functions/meta/#storage_policies) view is -always blank in the current release. - -::: - -### TTL format - -Follow each setting with a duration value using one of these formats: - -- Long form: `3 DAYS`, `1 MONTH`, `2 YEARS` -- Short form: `3d`, `1M`, `2Y` - -Supported units: `HOUR`/`h`, `DAY`/`d`, `WEEK`/`W`, `MONTH`/`M`, `YEAR`/`Y`. -Both singular and plural forms are accepted. - -### Constraints - -- TTL values must be in ascending order: - `TO PARQUET <= DROP NATIVE <= DROP LOCAL <= DROP REMOTE` -- All TTL values must be positive — `0` is rejected -- Each setting can only appear once per statement -- The materialized view must have a designated timestamp and partitioning enabled -- If the materialized view has a TTL set, clear it with - `ALTER MATERIALIZED VIEW SET TTL 0` before setting a storage policy. Any - non-zero `SET TTL` value is rejected in Enterprise with - `TTL settings are deprecated, please, create a storage policy instead` -- `ENABLE` and `DISABLE` require a policy to exist on the view; both return an - error otherwise - -### Permissions - -Each operation requires a specific permission: - -| SQL command | Required permission | -|-------------|-------------------| -| `SET STORAGE POLICY` | `SET STORAGE POLICY` | -| `DROP STORAGE POLICY` | `REMOVE STORAGE POLICY` | -| `ENABLE STORAGE POLICY` | `ENABLE STORAGE POLICY` | -| `DISABLE STORAGE POLICY` | `DISABLE STORAGE POLICY` | - -## Examples - -Set a storage policy with multiple stages: - -```questdb-sql -ALTER MATERIALIZED VIEW trades_hourly SET STORAGE POLICY( - TO PARQUET 7 DAYS, - DROP NATIVE 14 DAYS, - DROP LOCAL 1 MONTH -); -``` - -Update only the Parquet conversion threshold: - -```questdb-sql -ALTER MATERIALIZED VIEW trades_hourly SET STORAGE POLICY(TO PARQUET 14d); -``` - -Temporarily suspend a policy: - -```questdb-sql -ALTER MATERIALIZED VIEW trades_hourly DISABLE STORAGE POLICY; -``` - -Re-enable it: - -```questdb-sql -ALTER MATERIALIZED VIEW trades_hourly ENABLE STORAGE POLICY; -``` - -Remove a policy entirely: - -```questdb-sql -ALTER MATERIALIZED VIEW trades_hourly DROP STORAGE POLICY; -``` - -Check active policies: - -```questdb-sql -SELECT * FROM storage_policies; -``` - -## See also - -- [Storage Policy concept](/docs/concepts/storage-policy/) -- [ALTER TABLE SET STORAGE POLICY](/docs/query/sql/alter-table-set-storage-policy/) -- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/) -- [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/) -- [`storage_policies`](/docs/query/functions/meta/#storage_policies) — system - view listing active policies -- [RBAC permissions](/docs/security/rbac/#permissions) — `SET`, `REMOVE`, - `ENABLE`, and `DISABLE STORAGE POLICY` permissions diff --git a/documentation/query/sql/alter-mat-view-set-ttl.md b/documentation/query/sql/alter-mat-view-set-ttl.md index 143eff9ed..6a103788a 100644 --- a/documentation/query/sql/alter-mat-view-set-ttl.md +++ b/documentation/query/sql/alter-mat-view-set-ttl.md @@ -8,18 +8,6 @@ description: Sets the [time-to-live](/docs/concepts/ttl/) (TTL) period on a materialized view, automatically dropping partitions older than the specified duration. -:::caution - -**QuestDB Enterprise: TTL is deprecated.** Enterprise rejects any non-zero -`SET TTL` with -`TTL settings are deprecated, please, create a storage policy instead`. Use -[`ALTER MATERIALIZED VIEW SET STORAGE POLICY`](/docs/query/sql/alter-mat-view-set-storage-policy/) -instead. `SET TTL 0` is still accepted, for clearing a pre-existing TTL before -attaching a storage policy. See [Storage Policy](/docs/concepts/storage-policy/) -for the Enterprise replacement. - -::: - ## Syntax ``` diff --git a/documentation/query/sql/alter-table-set-storage-policy.md b/documentation/query/sql/alter-table-set-storage-policy.md index 32e7571f1..5346af177 100644 --- a/documentation/query/sql/alter-table-set-storage-policy.md +++ b/documentation/query/sql/alter-table-set-storage-policy.md @@ -4,9 +4,7 @@ sidebar_label: SET STORAGE POLICY description: ALTER TABLE SET STORAGE POLICY SQL keyword reference documentation. --- -Sets, modifies, enables, disables, or removes a storage policy on a table. For -the equivalent operations on materialized views, see -[ALTER MATERIALIZED VIEW SET STORAGE POLICY](/docs/query/sql/alter-mat-view-set-storage-policy/). +Sets, modifies, enables, disables, or removes a storage policy on a table. :::note @@ -24,7 +22,7 @@ a full overview. ```questdb-sql ALTER TABLE table_name SET STORAGE POLICY( [TO PARQUET ttl,] - [DROP NATIVE ttl,] + [TO REMOTE ttl,] [DROP LOCAL ttl,] [DROP REMOTE ttl] ); @@ -57,18 +55,20 @@ transition from native format to Parquet and eventually get removed: | Setting | Effect | |---------|--------| -| `TO PARQUET ` | Convert partition from native format to Parquet locally | -| `DROP NATIVE ` | Remove native binary files, keeping only the local Parquet copy | +| `TO PARQUET ` | Convert partition from native format to Parquet locally. The native files are removed and reads are served from the Parquet file | +| `TO REMOTE ` | _Reserved._ Will upload the partition to object storage when remote upload is supported | | `DROP LOCAL ` | Remove all local copies of the partition | | `DROP REMOTE ` | _Reserved._ Will remove the partition from object storage when remote upload is supported | :::info -`DROP REMOTE` is reserved syntax. It is rejected at SQL parse time with +`TO REMOTE` and `DROP REMOTE` are reserved syntax. They are rejected at SQL +parse time with `'TO REMOTE' is not supported yet` and `'DROP REMOTE' is not supported yet`. Automatic upload of Parquet files to object storage is not currently supported — storage policies operate locally -only. Because the clause cannot take effect, the `drop_remote` column in the -[`storage_policies`](/docs/query/functions/meta/#storage_policies) view is +only. Because these clauses cannot take effect, the `to_remote` and +`drop_remote` columns in the +[`storage_policies`](/docs/query/functions/meta/#storage_policies) view are always blank in the current release. ::: @@ -85,8 +85,10 @@ Both singular and plural forms are accepted. ### Constraints -- TTL values must be in ascending order: - `TO PARQUET <= DROP NATIVE <= DROP LOCAL <= DROP REMOTE` +- A drop stage may not fire before the write stage it depends on: + `TO PARQUET <= DROP LOCAL`, `TO REMOTE <= DROP LOCAL`, and + `DROP LOCAL <= DROP REMOTE`. `TO PARQUET` and `TO REMOTE` are independent of + each other - All TTL values must be positive — `0` is rejected - Each setting can only appear once per statement - The table must have a designated timestamp and partitioning enabled @@ -110,12 +112,11 @@ Each operation requires a specific permission: ## Examples -Set a storage policy with all three currently-supported stages: +Set a storage policy with both currently-supported stages: ```questdb-sql ALTER TABLE sensor_data SET STORAGE POLICY( TO PARQUET 3 DAYS, - DROP NATIVE 10 DAYS, DROP LOCAL 1 MONTH ); ``` @@ -161,7 +162,7 @@ CREATE TABLE 'sensor_data' ( ts TIMESTAMP, value DOUBLE ) timestamp(ts) PARTITION BY DAY -STORAGE POLICY(TO PARQUET 3 DAYS, DROP NATIVE 10 DAYS, DROP LOCAL 1 MONTH) WAL; +STORAGE POLICY(TO PARQUET 3 DAYS, DROP LOCAL 1 MONTH) WAL; ``` Stages that are not set are omitted from the output. @@ -169,7 +170,6 @@ Stages that are not set are omitted from the output. ## See also - [Storage Policy concept](/docs/concepts/storage-policy/) -- [ALTER MATERIALIZED VIEW SET STORAGE POLICY](/docs/query/sql/alter-mat-view-set-storage-policy/) - [CREATE TABLE](/docs/query/sql/create-table/) — `STORAGE POLICY` clause at table creation - [ALTER TABLE SET TTL](/docs/query/sql/alter-table-set-ttl/) — the TTL diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 4d1444671..0dd973953 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -21,16 +21,13 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] viewName AS [ ( ] query [ ) ] [ TIMESTAMP ( columnRef ) ] [ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) - [ TTL n timeUnit - | STORAGE POLICY ( policyStage [, policyStage ...] ) ] ] + [ TTL n timeUnit ] ] [ OWNED BY ownerName ] ``` Where: - `interval`: Duration like `1m`, `10m`, `1h`, `1d` - `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` -- `policyStage`: `TO PARQUET duration | DROP NATIVE duration | DROP LOCAL duration | DROP REMOTE duration` - (Enterprise only; all stages optional; durations must be positive and in ascending order) - `query`: Must contain `SAMPLE BY` or time-based `GROUP BY` ## Parameters @@ -46,7 +43,6 @@ Where: | `TIMESTAMP` | Designate timestamp column for the view | | `PARTITION BY` | Partitioning unit for view storage | | `TTL` | Retention period for view data | -| `STORAGE POLICY` | Partition lifecycle automation (Enterprise) — mutually exclusive with `TTL` | | `OWNED BY` | Assign ownership (Enterprise) | ## Rules and defaults @@ -282,45 +278,6 @@ Time units: `HOURS`, `DAYS`, `WEEKS`, `MONTHS`, `YEARS` The view's TTL is independent of the base table's TTL. See [TTL documentation](/docs/concepts/ttl/) for details. -:::note - -In QuestDB Enterprise, `TTL` is deprecated — `CREATE MATERIALIZED VIEW ... TTL` -is rejected with `TTL settings are deprecated, please, create a storage policy -instead`. Use `STORAGE POLICY` instead. If a legacy materialized view has a TTL -set, clear it with `ALTER MATERIALIZED VIEW SET TTL 0` before setting a storage -policy. - -::: - -## Storage Policy - -:::note - -Storage policies are available in **QuestDB Enterprise** only. - -::: - -A [storage policy](/docs/concepts/storage-policy/) automates the partition -lifecycle by defining when partitions are converted to Parquet locally, when -native data is removed, and when local copies are dropped. Place the -`STORAGE POLICY(...)` clause after `PARTITION BY`: - -```questdb-sql title="With storage policy (Enterprise)" -CREATE MATERIALIZED VIEW trades_hourly AS ( - SELECT timestamp, symbol, avg(price) AS avg_price FROM trades SAMPLE BY 1h -) PARTITION BY DAY - STORAGE POLICY(TO PARQUET 7d, DROP NATIVE 14d); -``` - -A storage policy supports up to four settings: `TO PARQUET`, `DROP NATIVE`, -`DROP LOCAL`, and `DROP REMOTE`. All are optional, all TTL values must be -positive, and they must be in ascending order. `DROP REMOTE` is reserved -syntax and is currently rejected at SQL parse time with -`'DROP REMOTE' is not supported yet`. - -To modify a storage policy after creation, see -[ALTER MATERIALIZED VIEW SET STORAGE POLICY](/docs/query/sql/alter-mat-view-set-storage-policy/). - ## Complete example Putting it all together: @@ -434,7 +391,6 @@ GRANT DROP MATERIALIZED VIEW ON trades_hourly TO user1; | `base table does not exist` | Referenced table doesn't exist | | `query is not supported` | Query doesn't meet constraints (missing SAMPLE BY, uses FILL, etc.) | | `permission denied` | Missing required permission (Enterprise) | -| `TTL settings are deprecated, please, create a storage policy instead` | `TTL` clause used in QuestDB Enterprise — use `STORAGE POLICY` instead | ## See also @@ -442,5 +398,4 @@ GRANT DROP MATERIALIZED VIEW ON trades_hourly TO user1; - [REFRESH MATERIALIZED VIEW](/docs/query/sql/refresh-mat-view/) - [DROP MATERIALIZED VIEW](/docs/query/sql/drop-mat-view/) - [ALTER MATERIALIZED VIEW SET REFRESH](/docs/query/sql/alter-mat-view-set-refresh/) -- [ALTER MATERIALIZED VIEW SET STORAGE POLICY](/docs/query/sql/alter-mat-view-set-storage-policy/) - [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/) diff --git a/documentation/query/sql/create-table.md b/documentation/query/sql/create-table.md index 22048cb33..f866f25d2 100644 --- a/documentation/query/sql/create-table.md +++ b/documentation/query/sql/create-table.md @@ -70,12 +70,13 @@ TABLE [IF NOT EXISTS] tableName Where `policyStage` is one of: ``` -TO PARQUET duration | DROP NATIVE duration | DROP LOCAL duration | DROP REMOTE duration +TO PARQUET duration | TO REMOTE duration | DROP LOCAL duration | DROP REMOTE duration ``` -Stages are Enterprise-only, all optional, and their durations must be positive -and in ascending order. `TTL` and `STORAGE POLICY` are mutually exclusive — see -[Storage Policy](#storage-policy). +Stages are Enterprise-only, all optional, and their durations must be positive. +A drop stage may not precede the write it depends on (`TO PARQUET` and +`TO REMOTE` before `DROP LOCAL`; `DROP LOCAL` before `DROP REMOTE`). `TTL` and +`STORAGE POLICY` are mutually exclusive — see [Storage Policy](#storage-policy). ```questdb-sql title="Create from another table's structure (CREATE TABLE LIKE)" CREATE TABLE tableName (LIKE sourceTableName); @@ -277,9 +278,9 @@ Storage policies are available in **QuestDB Enterprise** only. ::: A [storage policy](/docs/concepts/storage-policy/) automates the partition -lifecycle by defining when partitions are converted to Parquet locally, when -native data is removed, and when local copies are dropped. Place the -`STORAGE POLICY(...)` clause after `PARTITION BY`: +lifecycle by defining when partitions are converted to Parquet locally and +when local copies are dropped. Place the `STORAGE POLICY(...)` clause after +`PARTITION BY`: ```questdb-sql title="With storage policy (Enterprise)" CREATE TABLE trades ( @@ -289,14 +290,18 @@ CREATE TABLE trades ( amount DOUBLE ) TIMESTAMP(timestamp) PARTITION BY DAY -STORAGE POLICY(TO PARQUET 3d, DROP NATIVE 10d, DROP LOCAL 1M) +STORAGE POLICY(TO PARQUET 3d, DROP LOCAL 1M) WAL; ``` -A storage policy supports up to four settings: `TO PARQUET`, `DROP NATIVE`, -`DROP LOCAL`, and `DROP REMOTE`. All are optional, all TTL values must be -positive, and they must be in ascending order. `DROP REMOTE` is reserved -syntax and is currently rejected at SQL parse time with +A storage policy supports up to four settings: `TO PARQUET`, `TO REMOTE`, +`DROP LOCAL`, and `DROP REMOTE`. All are optional and all TTL values must be +positive. A drop stage may not precede the write it depends on (`TO PARQUET` +and `TO REMOTE` before `DROP LOCAL`; `DROP LOCAL` before `DROP REMOTE`), while +`TO PARQUET` and `TO REMOTE` are independent. Converting a partition to Parquet +removes its native files and serves reads from the Parquet file. `TO REMOTE` +and `DROP REMOTE` are reserved syntax and are currently rejected at SQL parse +time with `'TO REMOTE' is not supported yet` and `'DROP REMOTE' is not supported yet`. To modify a storage policy after table creation, see diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index b71252116..241caecc5 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -157,7 +157,7 @@ CREATE TABLE 'sensor_data' ( ts TIMESTAMP, value DOUBLE ) timestamp(ts) PARTITION BY DAY -STORAGE POLICY(TO PARQUET 3 DAYS, DROP NATIVE 10 DAYS, DROP LOCAL 1 MONTH) WAL; +STORAGE POLICY(TO PARQUET 3 DAYS, DROP LOCAL 1 MONTH) WAL; ``` Stages that are not configured on the policy are omitted from the clause. A diff --git a/documentation/sidebars.js b/documentation/sidebars.js index fe0ca917e..995512908 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -310,7 +310,6 @@ module.exports = { "query/sql/alter-mat-view-resume-wal", "query/sql/alter-mat-view-set-refresh", "query/sql/alter-mat-view-set-refresh-limit", - "query/sql/alter-mat-view-set-storage-policy", "query/sql/alter-mat-view-set-ttl", ], },