test: add advanced dictionary test#23483
Conversation
|
cc @kosiew |
There was a problem hiding this comment.
@Rich-T-kid
Thanks for working on this. I think the regression test is headed in the right direction, but I don't think it currently exercises the case we want to protect against. In addition, I have one small suggestion to simplify the coverage.
| FROM (VALUES ('west'), ('east'), ('west'), (NULL)) AS t(column1) | ||
| ), | ||
| second_batch AS ( | ||
| SELECT arrow_cast(column1, 'Dictionary(Int16, LargeUtf8)') AS region |
There was a problem hiding this comment.
I don't think these new regression cases are exercising dictionary grouping at the aggregation boundary.
Because the UNION inputs use different dictionary key/value types, for example Dictionary(Int32, Utf8) unioned with Dictionary(Int16, LargeUtf8) here, the physical plan coerces both branches back to LargeUtf8 before UnionExec and AggregateExec: CAST(CAST(column1 AS Dictionary(...)) AS LargeUtf8).
That means the GROUP BY sees plain strings, so an implementation that incorrectly hashed dictionary key ids would still pass these tests.
Could you construct the inputs using the same dictionary type while varying the dictionary value order instead? Alternatively, if there's another approach, could you verify through the plan or test setup that AggregateExec is actually consuming separate dictionary arrays with different dictionaries?
| statement ok | ||
| DROP TABLE dict_count_distinct; | ||
|
|
||
| # overlapping values with different key assignments across batches |
There was a problem hiding this comment.
Once the construction is fixed, I think one focused cross input dictionary key regression test would be enough here. A short comment noting that the plan was checked to preserve the dictionary type through the UNION would also make the intent clear. The current three cases are quite similar, and a single targeted case would be a bit easier to maintain.
2 participants
Which issue does this PR close?
follow up PR for #23280
Rationale for this change
see comment #23280 (comment)
What changes are included in this PR?
Adds SQL logic tests to verify that GROUP BY on dictionary-encoded columns resolves on values rather than raw dictionary key integers. The tests use UNION ALL between independently-encoded subqueries (mixing Int32/Int16/Int8 key types and Utf8/LargeUtf8 value types)
Are these changes tested?
the test cover
Are there any user-facing changes?
no