Conversation
Summary:
The macOS unittest job has been hitting its timeout for several runs in a
row with the progress bar frozen partway through pytest. Add
`pytest-timeout` so a stuck test fails with a nodeid and per-thread stack
trace, and set `faulthandler_timeout=180` so every xdist worker dumps its
threads every 3 minutes while tests are still running -- this surfaces the
hung test as it develops, not just at termination.
Diagnostic runs identified the hang fingerprint: tests stuck inside
`torch._inductor.package._package.__call__` (line 736), the AOTI-packaged
`.so` invocation. The same stack was observed for tests in
`examples/models/llama3_2_vision/{preprocess,vision_encoder}` and almost
certainly affects the other AOTI call sites we ship. Local M1 reruns of
the affected tests with the CI-pinned torch wheel all pass quickly, so
the hang is CI-environment-specific (suspect: AOTI dlopen under heavy
xdist + coverage contention on the macos-m1-stable runner).
Skip every AOTI-invoking test on macOS CI only, via
`unittest-macos-cmake.sh` -- linux, windows, and local dev continue to
run them. The skip covers the three vision tests (whole files) plus the
specific `*_aoti` methods in `extension/llm/modules/test/test_*.py`.
Job timeout drops back to 30 minutes for fast iteration; pytest
`--timeout=1500` gives any single test 25 minutes before it is treated
as hung.
Co-Authored-By: Claude <noreply@anthropic.com>
Summary: Previous experiment used `-n 1` which still spawns an xdist worker process that buffers all output. The CI logs showed 462 tests' worth of progress dots flushed at a single timestamp, making it impossible to identify which test was hanging. Switch to `-p no:xdist` so tests run in the main process with unbuffered output, and add `-v` so each test name prints as it starts/completes. Combined with `faulthandler_timeout=180`, this will show exactly which test is running when the hang occurs. Faulthandler dumps from two prior CI runs pointed to the same test position (~26% through the suite) but caught different XNNPACK passes mid-retrace (FuseBatchNormPass in one, RemoveRedundantCopyPass in the other). The common pattern is `super().call()` retracing large delegate subgraphs — suspect is `test_all_models_with_recipes` running ResNet50 / ViT / DeepLabV3 through the full XNNPACK pass pipeline. Co-Authored-By: Claude <noreply@anthropic.com>
Summary: Three CI runs with AOTI tests enabled all showed the same pattern: tests pass normally up to ~55%, then 38-42 minutes of complete silence before job cancellation. Faulthandler cannot fire during the hang (no dump appears), confirming the thread is blocked in native C/C++ code — consistent with the AOTI `dlopen` / compiler invocation path. Re-introduce the AOTI skips from the first diagnostic commit. If the job completes without timeout, this confirms the AOTI tests are the sole cause of the hang. Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced May 29, 2026
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19880
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New Failure, 268 PendingAs of commit a59a5bf with merge base 88faab2 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Three CI runs with AOTI tests enabled all showed the same pattern:
tests pass normally up to ~55%, then 38-42 minutes of complete silence
before job cancellation. Faulthandler cannot fire during the hang
(no dump appears), confirming the thread is blocked in native C/C++
code — consistent with the AOTI
dlopen/ compiler invocation path.Re-introduce the AOTI skips from the first diagnostic commit. If the
job completes without timeout, this confirms the AOTI tests are the
sole cause of the hang.
Co-Authored-By: Claude noreply@anthropic.com
Stack created with Sapling. Best reviewed with ReviewStack.