FEAT: Add stress test pipeline and migrate it to new 1ES Pool/Images#432
FEAT: Add stress test pipeline and migrate it to new 1ES Pool/Images#432gargsaumya wants to merge 4 commits intomainfrom
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
gargsaumya
changed the title
gargsaumya
changed the title
There was a problem hiding this comment.
Pull request overview
Adds a new scheduled/manual OneBranch pipeline to run stress testing and introduces a new multi-threaded stress test suite for mssql-python aimed at validating pooling and concurrency behavior under sustained and high-concurrency load.
Changes:
- Added a new
tests/test_020_multithreaded_stress.pymodule with multi-threaded stress tests and a helper runner/metrics models. - Added
OneBranchPipelines/stress-test-pipeline.ymlto run the stress suite on Windows (LocalDB) and Linux (Docker SQL Server) on a daily schedule or manually.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 12 comments.
| File | Description |
|---|---|
| tests/test_020_multithreaded_stress.py | New concurrency-focused stress tests with a multi-threaded runner and workload scenarios (matrix, sustained load, lifecycle, heavy queries). |
| OneBranchPipelines/stress-test-pipeline.yml | New scheduled pipeline to provision DBs (LocalDB/Docker) and execute/publish stress + performance stress tests on Windows and Linux agents. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
gargsaumya
force-pushed
the
saumya/stress-tests-gh
branch
from
a62b99a to
a7ac237
Compare
gargsaumya
force-pushed
the
saumya/stress-tests-gh
branch
from
a7ac237 to
96841c3
Compare
| This is an extreme stress test to find the breaking point. | ||
| Expects some failures but tests for graceful degradation. | ||
| """ | ||
| num_threads = 1000 |
There was a problem hiding this comment.
1000 threads is really really large and extremely unrealistic.
Let's use something pragmatic pragmatic like 2 times the processor core count.
|
|
||
| assert not result.hung, f"Test hung: {num_threads} threads, pooling={pooling}" | ||
|
|
||
| # Adaptive expectations based on thread count |
There was a problem hiding this comment.
An enhancement to the comment would be great. It should explain how adaptive nature is working. And why it is the right thing to do.
| after = get_resource_usage() | ||
| print(f"After stress RSS: {after['rss_mb']} MB") | ||
|
|
||
| mem_delta = after["rss_mb"] - baseline["rss_mb"] |
There was a problem hiding this comment.
unit of measurement in variable name is useful. like mem_delta_in_mb
| Tests with JOINs, aggregations, and larger result sets. | ||
| """ | ||
| complex_query = """ | ||
| SELECT TOP 50 |
There was a problem hiding this comment.
how about making this select * instead. You will get more results and each thread will be busy for longer.
With 50 rows, even if sql is busy, we wont be keeping the driver busy
|
|
||
| # Check for excessive memory growth (potential leak) | ||
| # Allow up to 100MB growth for long test | ||
| assert mem_growth < 100, f"Potential memory leak: {mem_growth:.1f}MB growth" |
There was a problem hiding this comment.
unit of measure in var name is going to be helpful
|
|
||
| This tests for memory leaks and resource exhaustion over time. | ||
| """ | ||
| num_threads = 20 |
There was a problem hiding this comment.
Can we have a test which uses normal thread count i.e. threads = Processor cores. I am sure we can get the processor core from some python runtime env var.
While we are stressing the driver, with high thread count, it would be good to have a test which works in "normal" conditions also. But keeping each thread busy for longer.
…ss.py and fix Black formatting
gargsaumya
force-pushed
the
saumya/stress-tests-gh
branch
from
1673b0b to
51d6f81
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
2 participants
Work Item / Issue Reference
Summary
This pull request adds a comprehensive suite of multi-threaded stress tests for the
mssql-pythonpackage. The new tests cover a wide range of concurrency scenarios, sustained load, pool lifecycle, and heavy query scaling to ensure robust performance and stability under high-load conditions. The tests are cross-platform, feature deterministic shutdowns, and include adaptive validation of error rates.Concurrency and load testing:
test_concurrency_matrix) that runs queries with varying thread counts and connection pooling settings, validating error rates at different concurrency levels.test_sustained_load) that run multiple threads for extended durations to simulate real-world high-load situations.test_heavy_query_scaling) with large result sets and high thread counts to stress test the connection pool.Connection pool management:
test_pool_lifecycle_cycles) to repeatedly enable and disable connection pooling, verifying deterministic shutdown and resource cleanup.Test infrastructure and utilities:
ThreadResult,StressResult) and a runner class (MultiThreadedQueryRunner) for collecting detailed metrics and managing