acc: run spark_python_task locally#5648
Open
pietern wants to merge 3 commits into
Open
Conversation
Add an executeSparkPythonTask branch to the testserver's JobsRunNow so a job whose single task is a spark_python_task executes its python_file in a uv-created venv (with per-cluster venv caching, mirroring the wheel and notebook helpers) and captures stdout into JobRunOutputs. This lets the spark_python_task cluster run test run against the in-process fake instead of requiring real DBR, so it is flipped to Local = true while still running in cloud. Co-authored-by: Isaac
Contributor
Approval status: pending
|
Local execution is additive: retain CloudSlow and RunsOnDbr so the test still exercises real cluster execution on cloud and DBR, only adding Local = true. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: d8d4a39
20 interesting tests: 13 SKIP, 7 KNOWN
Top 20 slowest tests (at least 2 minutes):
|
bundle/run/basic's job is a spark_python_task; now that the testserver executes it, `bundle run` prints the script's stdout. Regenerate the golden to include it. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
executeSparkPythonTaskto the testserver sospark_python_taskruns against the local fake like the existing wheel/notebook tasks.The test gains
Local = truewhile keepingCloudSlowandRunsOnDbr, so real cluster execution on cloud and DBR is preserved.bundle/run/basic's job is also aspark_python_task, so its golden is updated to include the script output thatbundle runnow prints.This pull request and its description were written by Isaac.