[Prototype] Bigtable

[Kbhat1]

Describe your changes and provide context

Testing performed to validate your change

---

Note

Medium Risk
Adds an optional fallback path for pruned Get/Has state-store reads to query external Scylla/Cassandra or Bigtable, plus new ingestion tooling and cloud/db dependencies; misconfiguration or backend issues could affect historical query behavior.

Overview
Adds prototype historical-state offload support so SS point reads (Get/Has) for versions below local retention can fall back to an external store (Scylla/Cassandra or Bigtable), wrapped via historical.FallbackStateStore and enabled through new [state-store] TOML/flag config (mutually exclusive backends).

Introduces a new Kafka-based offload consumer (sei-db/state_db/ss/offload/consumer) with configurable sinks for Scylla and Bigtable, including schema/examples and retry/batching/backpressure behavior, and exports offload.NewSASLMechanism so external consumers share the producer auth path.

Extends SS config structs/templates/tests for the new historical-offload settings and adds new dependencies (cloud.google.com/go/bigtable, github.com/gocql/gocql, golang.org/x/oauth2).

^{Reviewed by Cursor Bugbot for commit 66f8c9f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	May 15, 2026, 9:24 PM

[codecov]

Codecov Report

❌ Patch coverage is 39.25234% with 715 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.14%. Comparing base (654d40b) to head (35f7143).
⚠️ Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/ss/offload/historical/bigtable.go	34.07%	194 Missing and 13 partials ⚠️
sei-db/state_db/ss/offload/consumer/consumer.go	0.00%	177 Missing ⚠️
sei-db/state_db/ss/offload/historical/scylla.go	49.29%	71 Missing and 1 partial ⚠️
sei-db/state_db/ss/offload/consumer/scylla.go	66.85%	48 Missing and 11 partials ⚠️
sei-db/state_db/ss/store.go	22.64%	39 Missing and 2 partials ⚠️
sei-db/state_db/ss/offload/historical/store.go	39.34%	32 Missing and 5 partials ⚠️
sei-db/state_db/ss/offload/consumer/bigtable.go	67.59%	28 Missing and 7 partials ⚠️
sei-db/state_db/ss/offload/consumer/config.go	39.28%	29 Missing and 5 partials ⚠️
sei-db/state_db/ss/offload/consumer/kafka.go	44.82%	31 Missing and 1 partial ⚠️
...ad/consumer/cmd/historical-scylla-consumer/main.go	0.00%	21 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3437      +/-   ##
==========================================
- Coverage   59.25%   59.14%   -0.12%     
==========================================
  Files        2110     2120      +10     
  Lines      174181   176106    +1925     
==========================================
+ Hits       103210   104155     +945     
- Misses      62044    62962     +918     
- Partials     8927     8989      +62     

Flag	Coverage Δ
sei-chain-pr	48.42% <39.25%> (?)
sei-db	70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
app/seidb.go	74.39% <100.00%> (+2.39%)	⬆️
sei-db/config/ss_config.go	100.00% <ø> (ø)
sei-db/state_db/ss/offload/kafka.go	59.37% <100.00%> (ø)
...ad/consumer/cmd/historical-scylla-consumer/main.go	0.00% <0.00%> (ø)
sei-db/state_db/ss/offload/consumer/kafka.go	44.82% <44.82%> (ø)
sei-db/state_db/ss/offload/consumer/config.go	39.28% <39.28%> (ø)
sei-db/state_db/ss/offload/consumer/bigtable.go	67.59% <67.59%> (ø)
sei-db/state_db/ss/offload/historical/store.go	39.34% <39.34%> (ø)
sei-db/state_db/ss/store.go	24.07% <22.64%> (-75.93%)	⬇️
sei-db/state_db/ss/offload/consumer/scylla.go	66.85% <66.85%> (ø)
... and 3 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cursor]

PR Summary

Medium Risk
Adds optional fallback reads to external Scylla/Bigtable stores and introduces new ingestion tooling and dependencies, which can affect state-store initialization and historical query behavior when enabled.

Overview
Adds a prototype historical-state offload path: when local state-store history has been pruned, point reads (Get/Has) can now optionally fall back to ScyllaDB/Cassandra or Google Bigtable via a new historical.FallbackStateStore wrapper.

Introduces a Kafka-based offload consumer (historical-scylla-consumer) with pluggable sinks for Scylla and Bigtable, plus schema/docs and example configs. Configuration is extended with new TOML/CLI fields for Scylla and Bigtable offload settings, and ss.NewStateStore now enforces that only one offload backend is configured at a time.

Dependencies are updated to include cloud.google.com/go/bigtable, github.com/gocql/gocql, and related indirect modules; Kafka SASL mechanism creation is exported as offload.NewSASLMechanism for reuse.

^{Reviewed by Cursor Bugbot for commit 35f7143. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ecdf5cc. Configure here.}