Skip to content

perf: skip lid blocks#407

Open
cheb0 wants to merge 2 commits into330-lid-block-size-configfrom
0-lid-block-skip
Open

perf: skip lid blocks#407
cheb0 wants to merge 2 commits into330-lid-block-size-configfrom
0-lid-block-skip

Conversation

@cheb0
Copy link
Copy Markdown
Collaborator

@cheb0 cheb0 commented Apr 24, 2026

Description

Summary

  • new frac version: v4
  • LID block ext1 now has firstLID and lastLID encoded
  • isContinued is only used for older frac versions
  • no legacy format TID adjusting for a new frac version
  • as previously, we always locate and load the first block eagerly but calling NextGeq allows to skip some blocks

Measurements

With 16k LID blocks, there are only 3 search requests (out of 50 I tested) where this feature improves cold query perf. All three search requests fit into "intersect dense and sparse column" type.

With 4k blocks there are more queries that speed up (even some aggs), but it's not sufficient to cover perf slowdown from using 4k LID block instead of 64k.

Query Type Env Ids Total cold, ms stddev cold (branch), ms stddev diff Trial 1 Trial 2 Trial 3 Trial 4
service:E and login:L and moduleName:S and (zone:A or zone:B or zone:C) and exists:message srch prod 8 0 17.04 ±1.64 13.91 ±0.54 -18.4% -18.5% -17.6% -19.9% -18.4%
request_host:huge AND remote_addr:X (found) srch lb 5 0 12.43 ±0.43 11.35 ±0.51 -8.7% -8.1% -5.4% -5.8% -8.7%
request_host:small AND response_status:200 hist lb 69 69 25.58 ±1.04 20.9 ±0.60 -18.3% -17% -17.9% -14.1% -18.3%

  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

@cheb0 cheb0 marked this pull request as ready for review April 24, 2026 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant