Add indexes on sp_action.rollout and rollout_group#3045
Open
clayly wants to merge 1 commit intoeclipse-hawkbit:masterfrom
Open
Add indexes on sp_action.rollout and rollout_group#3045clayly wants to merge 1 commit intoeclipse-hawkbit:masterfrom
clayly wants to merge 1 commit intoeclipse-hawkbit:masterfrom
Conversation
|
Thanks @clayly for taking the time to contribute to hawkBit! We really appreciate this. Make yourself comfortable while I'm looking for a committer to help you with your contribution. |
Rollout monitoring queries (existsByRolloutId, getStatusCountByRolloutId,
getStatusCountByRolloutGroupId) filter by rollout or rollout_group on
sp_action. The flyway baseline did not index either column, so Postgres
falls back to Seq Scan on every monitoring poll. With 16k action rows
this is meaningful — the group-count query takes ~500 ms without the
index and ~27 ms with it (Index Only Scan, Heap Fetches: 0).
Bench (16k rows, 1000 iter):
- WHERE tenant=? AND rollout_group=? 18.6x faster on PG
17.6x faster on YugabyteDB
- WHERE tenant=? AND rollout=? GROUP BY status 2.5x faster on PG
1.5x faster on YugabyteDB
Adds V1_20_2 sibling migrations for POSTGRESQL, H2, and MYSQL.
83390ff to
51c1316
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Several rollout-monitoring queries on
sp_action(existsByRolloutId,getStatusCountByRolloutId,getStatusCountByRolloutGroupId, etc.) filter byrolloutorrollout_group. The flyway baseline does not index either column, so Postgres falls back toSeq Scan, scanning the entiresp_actiontable on every monitoring poll.For a 16 k-device rollout this means 16 k action rows scanned per call, multiple times per second during dispatch and monitoring. Beyond a single rollout, the table grows over the lifetime of the deployment and the cost grows linearly.
Fix
Two composite indexes:
sp_idx_action_rollout_statuson(tenant, rollout, status)— covers both filter andGROUP BY statuspatterns; enables Index Only Scan with no heap fetchsp_idx_action_rollout_groupon(tenant, rollout_group)— coversrollout_group=?lookups (Hibernate-generated SQL always includes tenant via the multi-tenant filter)V1_20_2__action_rollout_indexesmigrations added for POSTGRESQL, H2, and MYSQL.Idempotency
Each migration is guarded so re-applying the SQL against a schema that already has the index is a no-op:
CREATE INDEX IF NOT EXISTSIF NOT EXISTSforCREATE INDEX, so the migration uses anINFORMATION_SCHEMAlookup withPREPARE/EXECUTE. MariaDB and MySQL behave identically here.This protects deployments that may already have created these indexes out-of-band (e.g. via a fork's repeatable migration, an operator hot-fix, or a re-run after a failed migration).
Performance evidence
Insert 16 000 rows into
sp_actiontied torollout=1, spread acrossrollout_group=1..17. Run the hot queries 1000 times each, before and after the indexes.PostgreSQL 16
WHERE tenant=? AND rollout=? GROUP BY statusWHERE tenant=? AND rollout_group=?countWHERE rollout=? LIMIT 1(exists)Plans switch from
Seq ScantoIndex Only Scan,Heap Fetches: 0. Buffer reads drop from 214 → 17 (12x).YugabyteDB (PostgreSQL-compatible YSQL)
WHERE tenant=? AND rollout=? GROUP BY statusWHERE tenant=? AND rollout_group=?countWHERE rollout=? LIMIT 1(exists)Plain PG syntax works on YB. The leading column gets HASH-partitioned across tablets by default, optimal for the equality-on-tenant access pattern.
Index Only Scanis supported on YB withHeap Fetches: 0.YB seq scans are 3–5x slower than PG seq scans (network round-trip per page across tablets), so the index is even more critical on a YugabyteDB deployment.
Test plan
EXPLAINconfirmsIndex Only Scanafter migration