Skip to content

fix(benchmarks): unblock codspeed runs on pytest 9#1647

Merged
collerek merged 2 commits into
masterfrom
fix/benchmarks-pytest9
May 4, 2026
Merged

fix(benchmarks): unblock codspeed runs on pytest 9#1647
collerek merged 2 commits into
masterfrom
fix/benchmarks-pytest9

Conversation

@collerek
Copy link
Copy Markdown
Collaborator

@collerek collerek commented May 4, 2026

Summary

  • Drop the no-op @pytest.mark.benchmark(...) decorator on the aio_benchmark fixture in benchmarks/conftest.py. Marks on fixtures never had any effect and pytest 9 promoted the deprecation to a hard PytestRemovedIn9Warning raised at conftest import time, so every benchmark run since the pytest 8.4.2 → 9.0.3 bump (build(deps-dev): bump pytest from 8.4.2 to 9.0.3 #1609) hung and got auto-cancelled at the 24h job timeout.
  • Prefix the rewritten name with "u_" in test_benchmark_update.py and test_benchmark_bulk_update.py. The assertion author.name != starting_first_name previously compared two random 5-char strings drawn from string.ascii_letters — over thousands of codspeed walltime iterations the ~1/311M per-call collision eventually surfaced (CI showed 'DpwEX' != 'DpwEX'). The prefix makes the new value structurally unable to match.

Without this, master has had no successful benchmark baseline since 53d2462 (2026-03-31), which is why the recent feature/flatten-fields PR (#1641) shows a phantom -35% regression and a "Different runtime environments detected" warning — codspeed is comparing against a 5-week-old run on different macro-runner hardware.

Test plan

  • pytest benchmarks/test_benchmark_update.py benchmarks/test_benchmark_bulk_update.py --codspeed passes locally on pytest 9.0.3 + pytest-codspeed 4.5.0
  • codspeed-benchmarks CI succeeds on this PR, producing a fresh master baseline once merged

collerek added 2 commits May 4, 2026 11:20
Both update benchmarks asserted that author.name differed after a
random rewrite, but the rewrite drew from the same alphabet/length
as the fixture-generated original (5 chars from string.ascii_letters),
so a collision (~1/311M per call) eventually surfaced over many
codspeed walltime iterations and CI runs. Prefix the rewritten name
with "u_" so the new value can never structurally equal the original.
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 4, 2026

Merging this PR will degrade performance by 34.64%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 49 (👁 49) regressed benchmarks
✅ 33 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
👁 WallTime test_count[250] 3 ms 3.6 ms -16.6%
👁 WallTime test_avg[500] 2.5 ms 3.4 ms -25.65%
👁 WallTime test_max[250] 2.4 ms 3.4 ms -27.16%
👁 WallTime test_making_and_inserting_models_in_bulk[10] 9.5 ms 10.6 ms -10.62%
👁 WallTime test_get_or_none[250] 2.6 ms 3.5 ms -25.38%
👁 WallTime test_get_or_create_when_create[10] 102.4 ms 119.7 ms -14.44%
👁 WallTime test_get_or_create_when_create[40] 413.6 ms 484.2 ms -14.57%
👁 WallTime test_saving_models_individually[20] 149.3 ms 167.6 ms -10.91%
👁 WallTime test_values[250] 5.5 ms 6.4 ms -14.2%
👁 WallTime test_min[500] 2.5 ms 3.4 ms -25.16%
👁 WallTime test_first[500] 2.5 ms 3.3 ms -25.67%
👁 WallTime test_get_or_none[1000] 2.7 ms 3.5 ms -24.14%
👁 WallTime test_deleting_all[250] 1.4 ms 2.2 ms -34.64%
👁 WallTime test_avg[250] 2.4 ms 3.5 ms -30.88%
👁 WallTime test_get_or_create_when_create[20] 209.4 ms 240.7 ms -13.01%
👁 WallTime test_get_one[500] 2.7 ms 3.5 ms -24.54%
👁 WallTime test_get_or_none[500] 2.7 ms 3.5 ms -24.56%
👁 WallTime test_values_list[250] 5.6 ms 6.5 ms -13.53%
👁 WallTime test_get_or_create_when_get[1000] 2.7 ms 3.5 ms -24.48%
👁 WallTime test_get_one[250] 2.7 ms 3.5 ms -24.18%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Comparing fix/benchmarks-pytest9 (695f4ea) with master (53d2462)

Open in CodSpeed

@collerek collerek enabled auto-merge (squash) May 4, 2026 09:59
@collerek collerek disabled auto-merge May 4, 2026 09:59
@collerek collerek merged commit 672db74 into master May 4, 2026
11 checks passed
@collerek collerek deleted the fix/benchmarks-pytest9 branch May 4, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant