[SPARK-56845] [K8S] Truncate ConfigMap names that exceed DNS subdomain limit

[shivadarshan-devadiga]

What changes were proposed in this pull request?

This PR fixes SPARK-56845 by adding a new configMapName(prefix, suffix) overload to KubernetesClientUtils that automatically handles ConfigMap names exceeding Kubernetes' 253-character DNS subdomain limit.

Why are the changes needed?

When spark.app.name is very long (>222 characters), the derived resourceNamePrefix combined with fixed suffixes (-hadoop-config, -krb5-file, -driver-podspec-conf-map) exceeds Kubernetes' DNS subdomain name limit of 253 characters. This causes driver submission to fail with the error: "must be no more than 253 characters".

The existing KubernetesClientUtils.configMapName(prefix) helper had truncation logic but was hardcoded to the -conf-map suffix and unused by these three call sites. This PR generalizes the solution to work with any suffix and migrates all three problematic call sites to use it.

Why fallback instead of truncation:

• Truncating the prefix could create name collisions when multiple applications have similar long names
• Using a unique ID (spark-) ensures each ConfigMap has a distinct, valid name
• A warning log alerts users when fallback occurs, maintaining observability

Does this PR introduce any user-facing change?

Yes, but only for edge cases with very long application names:

Previous behavior:

• Driver submission failed with "must be no more than 253 characters" error when spark.app.name was very long (>222 chars).

New behavior:

• ConfigMap names that would exceed 253 characters now automatically fall back to spark- format
• A warning is logged: "ConfigMap name '' exceeds the maximum length of 253 characters. Using fallback name '' instead."
• Driver submission succeeds instead of failing
• For applications with normal-length names (<222 chars), there is no change in behavior.

How was this patch tested?

Unit tests added in KubernetesClientUtilsSuite.scala:

• configMapName with custom suffix truncates correctly - Tests short names pass through, names at 253-char boundary, and fallback for names exceeding limit
• configMapName with different suffixes - Validates all three production suffixes (-hadoop-config, -krb5-file, -driver-podspec-conf-map)
• configMapName fallback when prefix+suffix exceeds limit - Tests edge cases with different prefix lengths
• configMapName backward compatibility with default suffix - Ensures existing configMapName(prefix) calls still work

Test execution:

Ran the full Kubernetes test suite:
Result: 348 tests passed, 0 failed (including the 4 new SPARK-56845 tests)

Test coverage:

• Normal operation (names within limit)
• Boundary condition (exactly 253 characters)
• Fallback behavior (names exceeding limit)
• All three production suffixes
• Backward compatibility with existing code

Was this patch authored or co-authored using generative AI tooling?

Test cases and documentation authored by: Claude

cc @TongWei1105 @LuciferYang @dongjoon-hyun

[shivadarshan-devadiga]

@LuciferYang Can you please check

[LuciferYang]

also cc @dongjoon-hyun