Skip to content

[shell-operator] fix: protect shared informer lifecycle#917

Open
fuldaxxx wants to merge 3 commits into
mainfrom
fix/kubeevents-factorystore-ctx-leak
Open

[shell-operator] fix: protect shared informer lifecycle#917
fuldaxxx wants to merge 3 commits into
mainfrom
fix/kubeevents-factorystore-ctx-leak

Conversation

@fuldaxxx

@fuldaxxx fuldaxxx commented Jul 2, 2026

Copy link
Copy Markdown
Member

Overview

Fix a shared-informer lifecycle bug in FactoryStore (pkg/kube_events_manager/factory.go).
A shared informer is a long-lived, store-owned resource, but its context used to be
derived from the first consumer that registered it. Cancelling that consumer's
context (e.g. when its monitor is stopped during a module reload) tore down the shared
informer for every other consumer still using it, silently freezing their snapshots
until the process was restarted. Ownership of the informer lifetime is moved to the
store, plus three supporting fixes (dead-factory detection, a narrower critical section,
and an un-swallowed error).

What this PR does / why we need it

Root cause. NewFactoryStore() did not receive a context, and add() derived the
factory context via context.WithCancel(ctx) from the caller's context. The chain
mgr.ctx → monitor.ctx → resourceInformer.ei.ctx → Start(ei.ctx) → add(ei.ctx) meant
factory.ctx was a child of the ei.ctx of whichever monitor created the factory first.
Because factories are shared across monitors with the same FactoryIndex
(GVR + namespace + selectors), stopping that first monitor cancelled the shared informer
for all remaining consumers, while the refcount (handlerRegistrations) was still > 0 so
the factory stayed cached. The surviving consumers were left attached to a dead informer.

This is a classic ownership bug: the lifetime of a shared, long-lived resource must be
owned by the store, not inherited from a transient consumer.

Impact. The failure is silent (no error, HasSynced() on a stopped-but-once-synced
informer returns true) and only recoverable by restarting the process. In
surfaced as an empty DexClient snapshot in user-authn: the hook feeds the desired
object list into values, Helm prunes anything not in the list, so an empty
to the dex-client-* Secret and OAuth2Client being deleted — breaking OIDC login to
the cluster. Mass module reloads (e.g. during an upgrade) are the trigger w

Special notes for your reviewer

Signed-off-by: Ruslan Gorbunov <ruslan.gorbunov@flant.com>
@fuldaxxx fuldaxxx requested review from ipaqsa and ldmonster July 2, 2026 13:10
@fuldaxxx fuldaxxx self-assigned this Jul 2, 2026
@fuldaxxx fuldaxxx added bug Something isn't working go Pull requests that update Go code labels Jul 2, 2026
Signed-off-by: Ruslan Gorbunov <ruslan.gorbunov@flant.com>
@fuldaxxx fuldaxxx marked this pull request as ready for review July 6, 2026 14:04
Signed-off-by: Ruslan Gorbunov <ruslan.gorbunov@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working go Pull requests that update Go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant