[SPARK-58114][PYTHON] Consolidate VALUE_NOT_ANY_OR_ALL into VALUE_NOT_ALLOWED#57236
Closed
zhengruifeng wants to merge 1 commit into
Closed
[SPARK-58114][PYTHON] Consolidate VALUE_NOT_ANY_OR_ALL into VALUE_NOT_ALLOWED#57236zhengruifeng wants to merge 1 commit into
zhengruifeng wants to merge 1 commit into
Conversation
HyukjinKwon
approved these changes
Jul 14, 2026
Member
|
Let's file a JIRA tho |
zhengruifeng
changed the title
zhengruifeng
added a commit
that referenced
this pull request
Jul 15, 2026
…_ALLOWED ### What changes were proposed in this pull request? Consolidate the specialized PySpark `VALUE_NOT_ANY_OR_ALL` error condition into the existing `VALUE_NOT_ALLOWED` condition. The classic and Spark Connect implementations of `DataFrame.dropna` now report the allowed values through the generic condition, and the shared test expectation is updated accordingly. ### Why are the changes needed? `VALUE_NOT_ANY_OR_ALL` is used only for `DataFrame.dropna(how=...)` and duplicates the existing generic condition for arguments restricted to a fixed set of values. Removing it reduces the number of narrowly scoped error conditions and aligns this validation with other PySpark APIs. ### Does this PR introduce _any_ user-facing change? Yes. Invalid values for `DataFrame.dropna(how=...)` now use the `VALUE_NOT_ALLOWED` error condition and its generic message. Valid calls and exception types are unchanged. ### How was this patch tested? The focused PySpark test was not run. The JSON error-condition file was parsed successfully, `git diff --check` passed, and the existing shared error assertion was updated for both classic and Spark Connect execution modes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) Closes #57236 from zhengruifeng/consolidate-value-not-any-or-all-dev4. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org> (cherry picked from commit 76c0688) Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
Contributor
Author
uros-b
reviewed
Jul 15, 2026
uros-b
left a comment
uros-b
left a comment
Member
There was a problem hiding this comment.
cc @marcuslin123 @HyukjinKwon, regarding #57250 (review)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Consolidate the specialized PySpark
VALUE_NOT_ANY_OR_ALLerror condition into the existingVALUE_NOT_ALLOWEDcondition. The classic and Spark Connect implementations ofDataFrame.dropnanow report the allowed values through the generic condition, and the sharedtest expectation is updated accordingly.
Why are the changes needed?
VALUE_NOT_ANY_OR_ALLis used only forDataFrame.dropna(how=...)and duplicates the existinggeneric condition for arguments restricted to a fixed set of values. Removing it reduces the
number of narrowly scoped error conditions and aligns this validation with other PySpark APIs.
Does this PR introduce any user-facing change?
Yes. Invalid values for
DataFrame.dropna(how=...)now use theVALUE_NOT_ALLOWEDerrorcondition and its generic message. Valid calls and exception types are unchanged.
How was this patch tested?
The focused PySpark test was not run. The JSON error-condition file was parsed successfully,
git diff --checkpassed, and the existing shared error assertion was updated for both classicand Spark Connect execution modes.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)