Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions documentation/changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,10 @@ This page tracks significant updates to the QuestDB documentation.

### Updated

- [Storage policy](/docs/concepts/storage-policy/) - Removed the `DROP NATIVE` stage; `TO PARQUET` now removes native files and serves reads from the Parquet file
- [SQL extensions and compatibility](/docs/concepts/deep-dive/sql-extensions/) - Rewritten as the canonical SQL dialect reference covering time-series clauses and joins, query syntax conveniences, storage extensions, PIVOT vs SQL Server/PostgreSQL, JSON and UNNEST, and a full inventory of standard SQL features not supported with QuestDB equivalents. Sidebar entry moved to the top of Query & SQL Reference
- [systemd service example](/docs/deployment/systemd/) - Corrected JVM argument ordering, switched to `UseParallelGC`, set `-Dcontainerized=false`, and added required setup paths
- [Materialized views](/docs/query/sql/create-mat-view/) - Removed storage policy support for materialized views; use `TTL` for retention instead
- [Aggregation functions](/docs/query/functions/aggregation/) - Added demo tags and updated examples with runnable queries
- [LATEST ON](/docs/query/sql/latest-on/) - Added demo tags to examples
- [JOIN](/docs/query/sql/join/) - Updated examples to use demo data
Expand Down
13 changes: 0 additions & 13 deletions documentation/concepts/materialized-views.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,16 +336,6 @@ CREATE MATERIALIZED VIEW trades_hourly AS (

The view's TTL is independent of the base table's TTL.

:::note

In QuestDB Enterprise, TTL is superseded by
[storage policies](/docs/concepts/storage-policy/). Use
[`STORAGE POLICY(...)`](/docs/query/sql/alter-mat-view-set-storage-policy/) on
a materialized view instead of `TTL` for graduated lifecycle management
(convert to Parquet, then drop).

:::

### Initial refresh

When created, materialized views start an **asynchronous full refresh**:
Expand Down Expand Up @@ -602,9 +592,6 @@ the replica's view was not fully up-to-date.
Sets the time limit for incremental refresh on a materialized view
- [`ALTER MATERIALIZED VIEW SET TTL`](/docs/query/sql/alter-mat-view-set-ttl/):
Sets the time-to-live (TTL) period on a materialized view
- [`ALTER MATERIALIZED VIEW SET STORAGE POLICY`](/docs/query/sql/alter-mat-view-set-storage-policy/):
Attaches a [storage policy](/docs/concepts/storage-policy/) to a
materialized view (QuestDB Enterprise)

- **Configuration**
- [Materialized views configs](/docs/configuration/materialized-views/):
Expand Down
175 changes: 82 additions & 93 deletions documentation/concepts/storage-policy.md

Large diffs are not rendered by default.

39 changes: 19 additions & 20 deletions documentation/getting-started/enterprise-quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -478,48 +478,48 @@ too — on a schedule you define. This supersedes plain TTL in Enterprise, where

### Migrating from TTL when upgrading from OSS

Tables and materialized views that were created in OSS keep their existing
`TTL` setting after you upgrade to Enterprise — no data is lost at upgrade
time. However, Enterprise rejects any **new** `TTL` changes (both
`CREATE ... TTL` and `ALTER ... SET TTL <non-zero>`) with:
Tables that were created in OSS keep their existing `TTL` setting after you
upgrade to Enterprise — no data is lost at upgrade time. However, Enterprise
rejects any **new** `TTL` changes on tables (both `CREATE TABLE ... TTL` and
`ALTER TABLE SET TTL <non-zero>`) with:

```
TTL settings are deprecated, please, create a storage policy instead
```

To move a legacy table or materialized view from `TTL` to a storage policy:
Materialized views are not affected: they continue to use `TTL` for retention
in Enterprise.

To move a legacy table from `TTL` to a storage policy:

1. **Clear the existing TTL** by setting it to `0`. This is the only `SET TTL`
value Enterprise accepts, and it is required before a storage policy can be
attached:
value Enterprise accepts on a table, and it is required before a storage
policy can be attached:

```questdb-sql title="Clear the legacy TTL"
ALTER TABLE trades SET TTL 0;
-- or, for a materialized view:
ALTER MATERIALIZED VIEW trades_hourly SET TTL 0;
```

2. **Attach a storage policy** that reproduces — and ideally extends — the
retention the TTL used to provide. A policy lets you keep data in Parquet
after you would previously have dropped it, so `DROP LOCAL` (or
`DROP NATIVE` if you don't want Parquet at all) is the stage that replaces
the old TTL horizon:
after you would previously have dropped it, so `DROP LOCAL` is the stage
that replaces the old TTL horizon:

```questdb-sql title="Replace a 1-month TTL with an equivalent policy"
ALTER TABLE trades SET STORAGE POLICY(
TO PARQUET 3 DAYS,
DROP NATIVE 10 DAYS,
DROP LOCAL 1 MONTH
);
```

If you want the policy to behave exactly like the old TTL (delete the
partition outright after the same interval), use a single-stage policy —
for example `STORAGE POLICY(DROP NATIVE 1 MONTH)` to match `TTL 1 MONTH`.
partition outright after the same interval, with no Parquet conversion),
use a single-stage policy — for example `STORAGE POLICY(DROP LOCAL 1 MONTH)`
to match `TTL 1 MONTH`.

Do this for every table and materialized view you want to keep managed
automatically. Tables without a storage policy retain their data indefinitely
once their legacy TTL has been cleared.
Do this for every table you want to keep managed automatically. Tables without
a storage policy retain their data indefinitely once their legacy TTL has been
cleared.

### Creating new tables with a storage policy

Expand All @@ -531,7 +531,7 @@ CREATE TABLE trades (
symbol SYMBOL,
price DOUBLE
) TIMESTAMP(ts) PARTITION BY DAY
STORAGE POLICY(TO PARQUET 3d, DROP NATIVE 10d, DROP LOCAL 1M)
STORAGE POLICY(TO PARQUET 3d, DROP LOCAL 1M)
WAL;
```

Expand All @@ -540,7 +540,6 @@ Or attach a policy to an existing table:
```questdb-sql title="Web Console - Set a storage policy on an existing table"
ALTER TABLE trades SET STORAGE POLICY(
TO PARQUET 3 DAYS,
DROP NATIVE 10 DAYS,
DROP LOCAL 1 MONTH
);
```
Expand Down
6 changes: 3 additions & 3 deletions documentation/high-availability/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,9 @@ replicated automatically and should be populated separately on each instance.
## Storage policies in a replicated cluster

[Storage policy](/docs/concepts/storage-policy/) definitions are stored in
WAL-backed system tables, so the policy itself — the `TO PARQUET`,
`DROP NATIVE`, and `DROP LOCAL` TTLs and the active/disabled status — is
replicated to every instance through the same WAL pipeline as user data.
WAL-backed system tables, so the policy itself — the `TO PARQUET` and
`DROP LOCAL` TTLs and the active/disabled status — is replicated to every
instance through the same WAL pipeline as user data.

Enforcement, however, runs **independently on each instance**. Parquet files
are produced locally and are not replicated; each node's storage policy job
Expand Down
10 changes: 5 additions & 5 deletions documentation/operations/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,13 +335,13 @@ To find your instance name, see [Backup instance name](#backup-instance-name).
### Interaction with storage policies

[Storage policies](/docs/concepts/storage-policy/) operate locally — they
convert partitions to Parquet in place and then drop native (and eventually
local Parquet) files on a schedule. Backups capture whatever is on local disk
at the time the backup runs:
convert partitions to Parquet in place (removing the native files) and
eventually drop the local Parquet files on a schedule. Backups capture
whatever is on local disk at the time the backup runs:

- Partitions still in native format are backed up as native files.
- Partitions that have been converted to Parquet (via the `TO PARQUET` stage,
after `DROP NATIVE` has fired) are backed up as Parquet files.
- Partitions that have been converted to Parquet (via the `TO PARQUET` stage)
are backed up as Parquet files.
- Once `DROP LOCAL` fires and removes a partition from local disk, subsequent
backups will no longer contain that partition — restoring an earlier backup
is the only way to recover it.
Expand Down
23 changes: 12 additions & 11 deletions documentation/query/functions/meta.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,8 +295,8 @@ Storage policies — and the `storage_policies` view — are available in
:::

`storage_policies` is a system view that lists every
[storage policy](/docs/concepts/storage-policy/) currently attached to a table
or materialized view. Query it like any other table:
[storage policy](/docs/concepts/storage-policy/) currently attached to a table.
Query it like any other table:

```questdb-sql
SELECT * FROM storage_policies;
Expand All @@ -306,9 +306,9 @@ SELECT * FROM storage_policies;

| Column | Type | Description |
|--------|------|-------------|
| `table_dir_name` | _STRING_ | Directory name of the table or materialized view the policy is attached to. Matches the `table_dir_name` column in [`tables()`](#tables) / [`materialized_views`](#materialized_views). |
| `table_dir_name` | _STRING_ | Directory name of the table the policy is attached to. Matches the `table_dir_name` column in [`tables()`](#tables). |
| `to_parquet` | _STRING_ | TTL for the `TO PARQUET` stage (e.g. `72h`, `1m`). Blank when the stage is not configured. |
| `drop_native` | _STRING_ | TTL for the `DROP NATIVE` stage. Blank when the stage is not configured. |
| `to_remote` | _STRING_ | Reserved — always blank in the current release. The `TO REMOTE` clause is rejected at SQL parse time with `'TO REMOTE' is not supported yet`. The column is kept for forward compatibility. |
| `drop_local` | _STRING_ | TTL for the `DROP LOCAL` stage. Blank when the stage is not configured. |
| `drop_remote` | _STRING_ | Reserved — always blank in the current release. The `DROP REMOTE` clause is rejected at SQL parse time with `'DROP REMOTE' is not supported yet`. The column is kept for forward compatibility. |
| `status` | _CHAR_ | Policy status. `A` = active (the policy is being enforced), `D` = disabled (via [`ALTER TABLE DISABLE STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/)). |
Expand All @@ -329,14 +329,15 @@ SELECT * FROM storage_policies;
SELECT * FROM storage_policies;
```

| table_dir_name | to_parquet | drop_native | drop_local | drop_remote | status | last_updated |
|----------------|------------|-------------|------------|-------------|--------|--------------|
| trades~12 | 72h | 240h | 1m | | A | 2025-01-15T10:30:00.000000Z |
| metrics~18 | 168h | | | | D | 2025-01-14T09:15:42.000000Z |
| table_dir_name | to_parquet | to_remote | drop_local | drop_remote | status | last_updated |
|----------------|------------|-----------|------------|-------------|--------|--------------|
| trades~12 | 72h | | 1m | | A | 2025-01-15T10:30:00.000000Z |
| metrics~18 | 168h | | | | D | 2025-01-14T09:15:42.000000Z |

The first row is a policy with three active stages (3-day Parquet conversion,
10-day native drop, 1-month local drop) and is currently enforced. The second
row has only the `TO PARQUET` stage set and has been temporarily disabled.
The first row is a policy with two active stages (3-day Parquet conversion and
1-month local drop) and is currently enforced. The second row has only the
`TO PARQUET` stage set and has been temporarily disabled. The `to_remote` and
`drop_remote` columns are reserved and always blank in the current release.

## table_columns

Expand Down
162 changes: 0 additions & 162 deletions documentation/query/sql/alter-mat-view-set-storage-policy.md

This file was deleted.

12 changes: 0 additions & 12 deletions documentation/query/sql/alter-mat-view-set-ttl.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,6 @@ description:
Sets the [time-to-live](/docs/concepts/ttl/) (TTL) period on a materialized
view, automatically dropping partitions older than the specified duration.

:::caution

**QuestDB Enterprise: TTL is deprecated.** Enterprise rejects any non-zero
`SET TTL` with
`TTL settings are deprecated, please, create a storage policy instead`. Use
[`ALTER MATERIALIZED VIEW SET STORAGE POLICY`](/docs/query/sql/alter-mat-view-set-storage-policy/)
instead. `SET TTL 0` is still accepted, for clearing a pre-existing TTL before
attaching a storage policy. See [Storage Policy](/docs/concepts/storage-policy/)
for the Enterprise replacement.

:::

## Syntax

```
Expand Down
Loading
Loading