Skip to content

Shutdown task loss prevention#2820

Open
Sushisource wants to merge 5 commits intomasterfrom
shutdown-task-loss-fix
Open

Shutdown task loss prevention#2820
Sushisource wants to merge 5 commits intomasterfrom
shutdown-task-loss-fix

Conversation

@Sushisource
Copy link
Copy Markdown
Member

What was changed

  • Introduces a ShutdownWorker RPC at the Worker level (previously only called from WorkflowWorker for sticky queues) covering all task queue types (workflow, activity, nexus), so the server
    knows the worker is going away and can stop dispatching tasks to it.
  • Adds a per-worker workerInstanceKey (UUID) sent on every poll request, allowing the server to correlate polls with the shutdown signal.
  • When the server advertises the workerPollCompleteOnShutdown capability, pollers wait for outstanding long-polls to complete naturally (server returns empty responses) instead of forcibly
    interrupting them. This prevents tasks that were dispatched to the worker just before shutdown from being lost.
  • On shutdownNow() (forceful/interrupt path), the ShutdownWorker RPC is still fired but not awaited — poller teardown proceeds immediately without blocking on the server response.
  • For older servers that don't support the capability, behavior is unchanged: polls are forcibly cancelled with a 1-second timeout.
  • Replaces the per-worker AtomicBoolean serverSupportsAutoscaling with a shared NamespaceCapabilities object populated from DescribeNamespace at factory startup.

Why?

Prevents possible task loss during the shutdown process. Equivalent to: temporalio/sdk-core#1122

Checklist

  1. Closes

  2. How was this tested:
    Added test, existing tests.

  3. Any docs updates needed?

@Sushisource Sushisource requested a review from a team as a code owner March 28, 2026 01:01
@Sushisource Sushisource force-pushed the shutdown-task-loss-fix branch from 4462301 to 52437d7 Compare March 28, 2026 01:01
@Sushisource Sushisource force-pushed the shutdown-task-loss-fix branch from 52437d7 to 97a247d Compare March 28, 2026 01:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant