Feat(optimizer): canonicalize internal query names#7580
Feat(optimizer): canonicalize internal query names#7580georgesittas wants to merge 4 commits intomainfrom
Conversation
georgesittas
force-pushed
the
jo/query_fingerprinting
branch
from
fd6502c to
bdfc9ca
Compare
georgesittas
force-pushed
the
jo/query_fingerprinting
branch
from
bdfc9ca to
778d33d
Compare
SQLGlot Integration Test ResultsComparing:
By Dialect
Overallmain: 113236 total, 112046 passed (pass rate: 98.9%), sqlglot version: sqlglot:jo/query_fingerprinting: 101037 total, 101037 passed (pass rate: 100.0%), sqlglot version: Transitions: Dialect pair changes: 0 previous results not found, 3 current results not found ✅ 38 test(s) passed |
georgesittas
force-pushed
the
jo/query_fingerprinting
branch
from
778d33d to
d9e7533
Compare
georgesittas
force-pushed
the
jo/query_fingerprinting
branch
from
bb2f924 to
667c02e
Compare
New optimizer pass that rewrites a query into a canonical structural form: data-contract names (base-table names/columns, top-level output aliases) are preserved, while internal names (table aliases, CTE/ subquery names, internal column aliases) are renamed to sequential _tN / _cN. Useful for query equivalence and deduplication. Handles set operations (UNION / UNION BY NAME), pivots, UDTFs, recursive CTEs, lateral/unnest, BigQuery whole-row struct refs, and case-folding semantics across dialects.
georgesittas
force-pushed
the
jo/query_fingerprinting
branch
from
667c02e to
fd51669
Compare
| from sqlglot._typing import E | ||
|
|
||
|
|
||
| def canonicalize_internal_names(expression: E) -> E: |
There was a problem hiding this comment.
Is this rule safe with DDLs like CREATE TABLE AS? I think we'd rename the selections so we'd break the data contract
There was a problem hiding this comment.
No, the intention was to make it work for Query instances. DDLs and other statements aren't taken into account. I can mention this in the docstring.
| next_table = name_sequence("_t") | ||
| next_column = name_sequence("_c") |
There was a problem hiding this comment.
What is the contract of this rule? Should it be that different queries get canonicalized differently? Because both SELECT * FROM foo and SELECT * FROM bar would be canonicalized to the same query selecting from _t0
There was a problem hiding this comment.
The assumption in this rule is that source tables are qualified with a schema and possibly a catalog. Meaning that for your example, foo and bar are treated as CTEs, and hence canonicalized. In the context of a larger query containing them, that is expected, because foo and bar would get different canonical names, and hence result in a different canonical query at the end.
New optimizer pass that rewrites a query into a canonical structural form: data-contract names (base-table names/columns, top-level output aliases) are preserved, while internal names (table aliases, CTE/subquery names, internal column aliases) are renamed to sequential
_tN/_cN. Useful for query equivalence and deduplication.Handles set operations (
UNION/UNION BY NAME), pivots,UDTFs, recursive CTEs,LATERAL/UNNEST, BigQuery whole-row struct refs, and case-folding semantics across dialects.