Skip to content

[FLUSS][Spark] Add set_cluster_configs and reset_cluster_configs procedures#3204

Open
XuQianJin-Stars wants to merge 1 commit intoapache:mainfrom
XuQianJin-Stars:spark-set-reset-cluster-configs
Open

[FLUSS][Spark] Add set_cluster_configs and reset_cluster_configs procedures#3204
XuQianJin-Stars wants to merge 1 commit intoapache:mainfrom
XuQianJin-Stars:spark-set-reset-cluster-configs

Conversation

@XuQianJin-Stars
Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3203

This PR adds set_cluster_configs and reset_cluster_configs procedures to the Spark connector, aligning it with the Flink connector which already provides a complete set of cluster configuration management procedures (get_cluster_configs, set_cluster_configs, reset_cluster_configs).

Currently, Spark users can only read cluster configurations via get_cluster_configs but cannot dynamically modify or reset them through SQL. This change closes that gap.

Brief change log

  • Added SetClusterConfigsProcedure to dynamically set cluster configuration values via CALL sys.set_cluster_configs(config_pairs => ARRAY('key1', 'value1', 'key2', 'value2')).
  • Added ResetClusterConfigsProcedure to reset cluster configurations to their default values via CALL sys.reset_cluster_configs(config_keys => ARRAY('key1', 'key2')).
  • Registered both new procedures in SparkProcedures.
  • Both procedures reuse the existing Admin.alterClusterConfigs() API with AlterConfigOpType.SET and AlterConfigOpType.DELETE respectively.
  • Updated the Spark procedures documentation with syntax, parameters, return types, and examples for both new procedures.

Tests

  • SetClusterConfigsProcedureTest:
    • set_cluster_configs: set a single configuration — sets a config and verifies it via get_cluster_configs
    • set_cluster_configs: set multiple configurations — sets multiple configs in one call
    • set_cluster_configs: empty config_pairs should fail — validates error on empty input
    • set_cluster_configs: odd number of config_pairs should fail — validates error on malformed input
  • ResetClusterConfigsProcedureTest:
    • reset_cluster_configs: set and then reset a configuration — sets a config, resets it, and verifies it's no longer DYNAMIC
    • reset_cluster_configs: reset multiple configurations — resets multiple configs in one call
    • reset_cluster_configs: empty config_keys should fail — validates error on empty input

API and Format

No. This change only adds new Spark SQL procedures. No existing API or storage format is affected.

Documentation

Yes. Updated website/docs/engine-spark/procedures.md with documentation for both set_cluster_configs and reset_cluster_configs procedures, including syntax, parameters, return types, examples, and notes.

@XuQianJin-Stars XuQianJin-Stars changed the title [FLUSS][Spark] Add set_cluster_configs and reset_cluster_configs procedures for Spark connector [FLUSS][Spark] Add set_cluster_configs and reset_cluster_configs procedures Apr 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Spark] Add set_cluster_configs and reset_cluster_configs procedures for Spark connector

1 participant