workflows/v2: defer confidential workflows when the TEE runner is unavailable#23149
Open
nadahalli wants to merge 4 commits into
Open
workflows/v2: defer confidential workflows when the TEE runner is unavailable#23149nadahalli wants to merge 4 commits into
nadahalli wants to merge 4 commits into
Conversation
…vailable When a HandlerInTee workflow subscribes on a node where the confidential- workflows capability is not available yet (e.g. still rolling out across the DON), the engine hard-failed init and error-stormed every sync tick. Now it holds the workflow and retries instead. - ConfidentialModule.Tee honors the enabledGate in the subscribe path (not just Execute), so a settings-disabled feature is inert rather than fatal. - teeProvider no longer caches an empty region set permanently (the old sync.Once): it re-queries until the capability reports regions, so the module self-heals once the capability comes up. - Engine init treats host.ErrRunnerUnavailable as a deferred (warn + retry) condition rather than a fatal 'Workflow Engine initialization failed'. Bumps chainlink-common for host.ErrRunnerUnavailable (chainlink-common #2259); depends on that PR, the bump will be re-pointed to its merged commit.
Contributor
|
✅ No conflicts with other open PRs targeting |
Contributor
|
I see you updated files related to
|
nadahalli
marked this pull request as ready for review
July 16, 2026 14:11
product-security-plaid-production
Bot
requested review from
ilija42,
pavel-raykov and
vyzaldysanchez
July 16, 2026 14:11
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When a confidential (
HandlerInTee) workflow subscribes on a node where theconfidential-workflows@1.0.0-alphacapability is not available yet (e.g. still rolling out across the DON), the engine hard-failed init with "cannot find a runner" and error-stormedWorkflow Engine initialization failedon every sync tick. This defers the workflow and retries instead.ConfidentialModule.Teenow honors theenabledGatein the subscribe path (not justExecute), so a settings-disabled feature is inert rather than fatal.teeProviderno longer caches an empty region set permanently (the oldsync.Once): it re-queries until the capability reports regions, so the module self-heals once the capability comes up.host.ErrRunnerUnavailable(from chainlink-common) as a deferred (warn + retry) condition rather than a fatalWorkflow Engine initialization failed.Why
On a mixed/rolling DON, the confidential-workflows capability is not on every node yet;
ConfidentialModule.providedTeesreports zero regions on those nodes, so subscribe fails and every confidential workflow error-storms init until the capability arrives. Worse, the oldsync.Oncekept the module stuck on the cached-empty result even after the capability recovered. This makes confidential workflows degrade gracefully and self-heal.Dependency
Depends on chainlink-common #2259 (adds
host.ErrRunnerUnavailable+ the runner-type-vs-unavailable distinction inrequirement_selecting_module). Thechainlink-commonbump here points at that PR branch commit and will be re-pointed to its merged commit before merge.Tests
core/services/workflows/v2suite green. AddedTeecases: disabled-gate short-circuits without querying the capability; empty region set is not cached and re-queries until regions appear (self-heal).