Skip to content

[SPARK-56270][PYTHON] Fix type hints and import issue for find_spark_home#55067

Open
gaogaotiantian wants to merge 1 commit intoapache:masterfrom
gaogaotiantian:fix-find-spark-home-type-hint
Open

[SPARK-56270][PYTHON] Fix type hints and import issue for find_spark_home#55067
gaogaotiantian wants to merge 1 commit intoapache:masterfrom
gaogaotiantian:fix-find-spark-home-type-hint

Conversation

@gaogaotiantian
Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  • Remove the type hint ignore for find_spark_home module and fill in the type hints
  • Remove the ImportError check and import_error_raised related logic

Why are the changes needed?

There's no reason we can't type hint this module so I just did it.

For the import part, it's impossible to trigger that path because find_spec("pyspark") will never trigger ImportError. It either returns a spec or None. I searched the history and the first time it was introduced was in #15659 . There were 2 branches and the one for python2 actually will trigger the ImportError. The python3 branch won't work but it might just be ignored.

In #28152 we added import_error_raised check and extra messages, but we only tested in python2, which would trigger that message. I confirmed that the message won't be printed if we can't find pyspark.

Now we print the message if we can't find SPARK_HOME.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Locally confirmed the message will be printed if pyspark can't be imported. Lint also passed locally.

Was this patch authored or co-authored using generative AI tooling?

No.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants