ClickHouse: Support scalar expressions in WITH clause by alrevuelta · Pull Request #2333 · apache/datafusion-sqlparser-rs

alrevuelta · 2026-05-07T10:43:43Z

ClickHouse's WITH clause supports a reversed-order form <expression> AS <identifier> (alongside, and freely interleaved with, traditional CTEs). Currently the parser hard-errors on the leading non-identifier token:

WITH 42 AS answer SELECT answer FROM t
-- Expected: identifier, found: 42

Changes

New WithItem enum (Cte(Cte) | Named { expr, alias }); With::cte_tables: Vec<Cte> becomes With::items: Vec<WithItem>.
New dialect flag Dialect::supports_with_clause_scalar_expression (default false, true for ClickHouseDialect only).
New parse_with_item routes each WITH-list element: when the leading token can't begin a CTE name, it parses <expr> AS <ident> directly; otherwise it tries the CTE form via maybe_parse and falls back. parse_cte itself is unchanged.

Tests

Added in tests/sqlparser_clickhouse.rs, covering: literal scalar (the issue's example), string literal, aggregate, scalar subquery, bare-identifier disambiguation, mixing scalar + CTE in one list, lambda, and a negative test confirming non-ClickHouse dialects still reject the form.

iffyio · 2026-05-07T11:35:22Z

+#[test]
+fn parse_with_clause_named_expression_unsupported_in_other_dialects() {
+    // The named-expression form is only enabled for ClickHouse; other
+    // dialects should still reject `WITH 42 AS answer …`.
+    let res = sqlparser::parser::Parser::parse_sql(
+        &GenericDialect {},
+        "WITH 42 AS answer SELECT answer FROM t",
+    );
+    assert!(res.is_err(), "expected parse error, got {res:?}");
+}


I think we can skip this, no need to test what the parser does for an invalid query for unsupported dialects

removed
3b86842#diff-b9a99857d69356e86daf74627f3d5da1f187d71e35dde75b0d924ed668ab1a43L1902

iffyio · 2026-05-07T11:35:42Z

+}
+
+#[test]
+fn parse_with_clause_named_expression_ast() {


we can merge this with the test above since they cover the same feature

merged
3b86842#diff-b9a99857d69356e86daf74627f3d5da1f187d71e35dde75b0d924ed668ab1a43L1887

iffyio · 2026-05-07T11:36:46Z

+#[test]
+fn parse_with_clause_named_expression() {
+    // Plain literal scalar.
+    clickhouse().verified_stmt("WITH 42 AS answer SELECT answer FROM t");


for the tests, lets use all_dialects_where instead of hardcoding it to clickhouse

done
3b86842#diff-b9a99857d69356e86daf74627f3d5da1f187d71e35dde75b0d924ed668ab1a43R1850

iffyio · 2026-05-07T11:40:19Z

+        // CTE form must start with an identifier. If the leading token
+        // can't begin one (e.g. `42`, `(SELECT …)`, `(x, y) -> …`), this
+        // is unambiguously the named-expression form.
+        if matches!(self.peek_token().token, Token::Word(_)) {
+            if let Some(cte) = self.maybe_parse(|p| p.parse_cte())? {
+                return Ok(WithItem::Cte(cte));
+            }
+        }


Can this be simplified to use peek_subquery_or_cte_start? e.g.

that we do if self.dialect.supports() and !self.peek_subquery_or_cte_start { parse expr } else { parse cte }

tbh, not sure i fully understand this. with that, will WITH 42 AS answer SELECT answer FROM t parse? peek_subquery_or_cte_start returns false on 42 (not (SELECT / (WITH), so we'd go to parse_cte, which then errors on 42 not being an identifier?

Yeah that would work right or? if we error on 42 its because the syntax was invalid and we expected a cte but got 42 instead?

right, i was mixing things up. can you confirm this WITH (SELECT max(x) FROM t) AS m SELECT m will work with your approach? i tried and seems that the first tokens (SELECT make peek_subquery_or_cte_start return true, which routes to parse_cte, but parse_cte immediately fails on ( since it expects an identifier.

I think the main goal with the comment was to simplify the cases a bit, so that ideally we have one branch handling the cte case, and another handling the cse case. Another way (there might be others) to accomplish that would be to try to parse a cte and if that fails fallback to cse e.g.

if let Some(cte) = self.maybe_parse(|parser| parser...)? else { return cte } else if self.supprots() { return self.parse_cse } else { self.expected("CTE or CSE") }

maybe above woul be straightforward in that the cases that are covered is clearer?

iffyio · 2026-05-07T11:40:52Z

+    /// Parse a single item in a `WITH` clause.
+    ///
+    /// In standard SQL this is always a CTE (`name [(cols)] AS (query)`).
+    /// Dialects that enable [`Dialect::supports_with_clause_scalar_expression`]
+    /// — currently only ClickHouse — also accept the reversed form
+    /// `<expression> AS <identifier>`, which can be freely interleaved with
+    /// CTEs in the same comma-separated list.


Suggested change

/// Parse a single item in a `WITH` clause.

///

/// In standard SQL this is always a CTE (`name [(cols)] AS (query)`).

/// Dialects that enable [`Dialect::supports_with_clause_scalar_expression`]

/// — currently only ClickHouse — also accept the reversed form

/// `<expression> AS <identifier>`, which can be freely interleaved with

/// CTEs in the same comma-separated list.

/// Parse a single item in a `WITH` clause.

fixed and minor renaming
3b86842#diff-fc6a8d66b8cb6bd48a119a70e489cbcef82e7f551e7cf08e8e972ef4e774ef49R14642

iffyio · 2026-05-07T11:46:09Z

+#[derive(Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash)]
+#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
+#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
+pub enum WithItem {


repr wise can we do something like?

pub enum WithExpression { Cte(...), Cse(ExprWithAlias), // can we reuse exprwithalias? }

fixed
3b86842#diff-ff51937f34d0076928ac31baffb41b26e4cbc0ec6dd8496105a7f61188f92ed1R776-R782

iffyio · 2026-05-07T11:47:04Z

+#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
+#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
+pub enum WithItem {
+    /// A traditional common table expression: `name [(cols)] AS [MATERIALIZED] (query)`.


Suggested change

/// A traditional common table expression: `name [(cols)] AS [MATERIALIZED] (query)`.

/// A common table expression.

fixed
3b86842#diff-ff51937f34d0076928ac31baffb41b26e4cbc0ec6dd8496105a7f61188f92ed1R776-R782

iffyio · 2026-05-07T11:47:36Z

+    /// `<expr> AS <alias>` — binds an expression (literal, scalar subquery,
+    /// lambda, …) to a name visible in the surrounding query.
+    ///
+    /// See ClickHouse's [common scalar expressions][1].
+    ///
+    /// [1]: https://clickhouse.com/docs/sql-reference/statements/select/with#common-scalar-expressions


Suggested change

/// `<expr> AS <alias>` — binds an expression (literal, scalar subquery,

/// lambda, …) to a name visible in the surrounding query.

///

/// See ClickHouse's [common scalar expressions][1].

///

/// [1]: https://clickhouse.com/docs/sql-reference/statements/select/with#common-scalar-expressions

/// A common scalar expression.

///

/// [Clickhouse]: https://clickhouse.com/docs/sql-reference/statements/select/with#common-scalar-expressions

iffyio · 2026-05-07T11:49:35Z

-    pub cte_tables: Vec<Cte>,
+    /// The items declared by this `WITH` clause: traditional CTEs and,
+    /// for dialects that support it, named expressions.
+    pub items: Vec<WithItem>,


Suggested change

pub items: Vec<WithItem>,

pub exprs: Vec<WithExpression>,

fixed
3b86842#diff-ff51937f34d0076928ac31baffb41b26e4cbc0ec6dd8496105a7f61188f92ed1R757-R758

iffyio · 2026-05-07T11:49:46Z

+    /// The items declared by this `WITH` clause: traditional CTEs and,
+    /// for dialects that support it, named expressions.


Suggested change

/// The items declared by this `WITH` clause: traditional CTEs and,

/// for dialects that support it, named expressions.

/// The expressions declared by this `WITH` clause.

fixed
3b86842#diff-ff51937f34d0076928ac31baffb41b26e4cbc0ec6dd8496105a7f61188f92ed1R757-R758

alrevuelta · 2026-05-11T11:17:54Z

@iffyio thanks for the comments, fixed.

there is only this that i dont fully understand if possible.

iffyio · 2026-05-13T04:46:05Z

+
+    // String literal scalar (from the ClickHouse docs).
+    dialects.verified_stmt(
+        "WITH '2019-08-01 15:23:00' AS ts_upper_bound SELECT * FROM hits \


for the line breaks can we use concat like e.g. here to format the sql strings?

fixed 4f69357

Allow WITH using scalars for ClickHouse dialect

6589675

iffyio reviewed May 7, 2026

View reviewed changes

Implement comment fixes

3b86842

iffyio reviewed May 13, 2026

View reviewed changes

use contat for queries

4f69357

	/// A traditional common table expression: `name [(cols)] AS [MATERIALIZED] (query)`.
	/// A common table expression.

		/// The items declared by this `WITH` clause: traditional CTEs and,
		/// for dialects that support it, named expressions.

	/// The items declared by this `WITH` clause: traditional CTEs and,
	/// for dialects that support it, named expressions.
	/// The expressions declared by this `WITH` clause.

Conversation

alrevuelta commented May 7, 2026

Changes

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alrevuelta commented May 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants