[tests] Skip PerformanceTest on slow CI machines using evaluation time#11249
Merged
simonrozsival merged 2 commits intomainfrom May 1, 2026
Merged
[tests] Skip PerformanceTest on slow CI machines using evaluation time#11249simonrozsival merged 2 commits intomainfrom
simonrozsival merged 2 commits intomainfrom
Conversation
When a performance test exceeds its expected time, check the MSBuild project evaluation duration from the binlog. If evaluation alone exceeds 500ms (normal is ~200-350ms), the CI machine is too slow for reliable performance measurements. Use Assert.Inconclusive() instead of Assert.Fail() so the test is marked as skipped rather than failed. Analysis of 10 recent CI failures showed evaluation times of 558-854ms on slow machines, while normal machines should be well under 500ms. Also refactored binlog reading so BinaryLog.ReadBuild() is called once per iteration, with the Build object passed to helper methods. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the MSBuild device integration performance tests to reduce false CI failures on slow machines by using MSBuild binlog project evaluation time as a “machine speed” signal.
Changes:
- Adds a slow-machine detector using the maximum
ProjectEvaluationduration from the binlog to decide betweenAssert.Inconclusive()vsAssert.Fail(). - Refactors binlog handling so
BinaryLog.ReadBuild()is done once per iteration and the parsedBuildis passed to duration helpers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers
added
the
ready-to-review
Member
Author
|
All the |
simonrozsival
approved these changes
May 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance tests commonly fail on slow CI machines, not due to actual regressions but because the machine is too slow for reliable measurements. This PR detects slow machines by checking the MSBuild project evaluation time from the binlog.
Changes
When a performance test exceeds its expected time, we now check the max
ProjectEvaluationduration from the binlog:Assert.Inconclusive()(test skipped, not failed)Assert.Fail()(genuine regression)Also refactored binlog reading so
BinaryLog.ReadBuild()is called once per iteration, with theBuildobject passed to helper methods instead of each method re-reading the file.Data
Analysis of 10 recent CI failure binlogs:
All failures had eval times 558–854ms. Normal machines should be ~200–350ms, so the 500ms threshold catches slow machines with comfortable margin.