Summary
Duplicate blocks accumulating in the block cache (chainN.blocks) can cause subgraphs to stop indexing. The subgraph remains health=healthy and active=true but processes 0 blocks/s. The only symptom is an increasing blocks_behind count.
The error is only visible at DEBUG level:
Block stream produced a non-fatal error, error: block stream error Block 14500860 does not contain hash
This repeats every ~7 minutes indefinitely. The subgraph never fails or enters an unhealthy state — it just stops advancing.
Root cause
The block ingestor stores both branches of a reorg in the block cache without cleaning up orphaned entries. Under certain conditions (not fully characterized — it does not happen at every duplicate block), a subgraph hits a block where the hash it expects no longer resolves, causing the "does not contain hash" error loop.
It is unclear what exact combination of factors triggers the issue — the presence of duplicate blocks is a necessary condition, but there may be additional factors (e.g., specific reorg depth, timing of block ingestion vs subgraph processing).
Impact observed (production, ~250 subgraphs)
| Chain |
Duplicate blocks |
Range checked |
Ratio |
| moonbeam |
2,400+ |
16,000 blocks |
~15% |
| mainnet (Ethereum PoS) |
1,382 |
full cache |
— |
- Blocked subgraphs can show very high
reorg_count (779,742 for the moonbeam case)
- The issue is silent — no WARN/ERROR at default log level, health stays healthy
- Observed on chains using RPC polling (not firehose)
Workaround
Remove duplicate blocks from the cache and perform a short rewind:
-- Find duplicate blocks
SELECT number, count(*) FROM chainN.blocks
GROUP BY number HAVING count(*) > 1;
# Remove duplicates (or truncate the whole chain cache)
graphman chain check-blocks <chain> by-range -f <from> -t <to> --delete-duplicates
# or: graphman chain truncate <chain>
# Then rewind the stuck subgraph with the correct block hash
graphman rewind --force --block-number <N> --block-hash <correct_hash> <deployment>
Related
Environment
Summary
Duplicate blocks accumulating in the block cache (
chainN.blocks) can cause subgraphs to stop indexing. The subgraph remainshealth=healthyandactive=truebut processes 0 blocks/s. The only symptom is an increasingblocks_behindcount.The error is only visible at DEBUG level:
This repeats every ~7 minutes indefinitely. The subgraph never fails or enters an unhealthy state — it just stops advancing.
Root cause
The block ingestor stores both branches of a reorg in the block cache without cleaning up orphaned entries. Under certain conditions (not fully characterized — it does not happen at every duplicate block), a subgraph hits a block where the hash it expects no longer resolves, causing the "does not contain hash" error loop.
It is unclear what exact combination of factors triggers the issue — the presence of duplicate blocks is a necessary condition, but there may be additional factors (e.g., specific reorg depth, timing of block ingestion vs subgraph processing).
Impact observed (production, ~250 subgraphs)
reorg_count(779,742 for the moonbeam case)Workaround
Remove duplicate blocks from the cache and perform a short rewind:
Related
Environment