gh-75572: Speed up test_xpickle #144393
gh-75572: Speed up test_xpickle #144393
Conversation
Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request.
serhiy-storchaka
added
tests
Lib/test/test_xpickle.py
Outdated
| worker = cls.worker | ||
| if worker is None: | ||
| target = os.path.join(os.path.dirname(__file__), 'xpickle_worker.py') | ||
| worker = subprocess.Popen([*python, target], |
There was a problem hiding this comment.
Why not calling start_worker() here?
There was a problem hiding this comment.
Oh, this is a remnant. I was not sure whether it is worth to move this code out into a method.
Lib/test/test_xpickle.py
Outdated
| cls.worker = worker | ||
|
|
||
| @classmethod | ||
| def close_worker(cls): |
There was a problem hiding this comment.
"stop_worker()" name would be better to have start/stop_worker methods.
| stdout=subprocess.PIPE, | ||
| stderr=subprocess.PIPE, | ||
| # For windows bpo-17023. | ||
| shell=is_windows) |
There was a problem hiding this comment.
You should use text=True to fix this code below:
_, stderr = worker.communicate()
raise RuntimeError(stderr)There was a problem hiding this comment.
But we do binary IO for stdin/stdout in normal case.
This code is only for troubleshooting. It is not executed when the tests pass.
vstinner
left a comment
vstinner
left a comment
There was a problem hiding this comment.
./python -m test test_xpickle -u xpickle,cpu -v command takes:
- main branch: 24 min 38 sec
- this PR: 1 min 24 sec
Well, it's much faster with this PR :-)
Note: I measured timings on a debug build (gcc -Og).
|
Note that you can run it only for the selected version, e.g. |
|
Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14. |
Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request. (cherry picked from commit 29acc08c8dad664cd5713cb392e5beba65724c10) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request. (cherry picked from commit 29acc08) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
|
GH-144403 is a backport of this pull request to the 3.14 branch. |
|
GH-144404 is a backport of this pull request to the 3.13 branch. |
2 participants
Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request.