feat: Implement TimeType support - Infrastructure - shuffle#4398
feat: Implement TimeType support - Infrastructure - shuffle#4398YutaLin wants to merge 10 commits into
Conversation
|
Hi @parthchandra |
|
Hi @YutaLin - could you address feedback and fix conflicts? I took a pass through and this is looking good. |
|
Moving this to draft for now. @YutaLin feel free to mark as ready-for-review once you have time to address feedback. |
|
Hi @parthchandra, thanks for the review, i've made some changes and add sql test cases for map/list. Could you review this again for me, do appreciate! cc @andygrove |
| -- Config: spark.sql.timeType.enabled=true | ||
| -- Config: spark.comet.native.shuffle.partitioning.roundrobin.enabled=true | ||
| -- ConfigMatrix: spark.comet.exec.shuffle.mode=native,jvm | ||
|
|
There was a problem hiding this comment.
Should we add a expect_fallback case with hash partitioning to ensure that fallback works correctly?
e.g. - SELECT /*+ REPARTITION_BY_RANGE(3, t) */ or DISTRIBUTE BY
|
Hi @parthchandra cc @andygrove |
3 participants
Which issue does this PR close?
Closes #4396.
Rationale for this change
Support shuffle for TimeType, include native and JVM columnar
What changes are included in this PR?
CometShuffleExchangeExec.scala, so the planner will offload a shuffle carrying a TimeType column instead of fallbackhash_funcs/utils.rsspark_unsafe/row.rs,spark_unsafe/list.rsshuffle_time.sqlfor test shuffle only(no aggregate)How are these changes tested?
run sql test in spark4.1, spark4.2