Skip to content

[#735] Do not roll back a concurrently adopted generation ID on aborted handshake#736

Open
vharseko wants to merge 2 commits into
OpenIdentityPlatform:masterfrom
vharseko:fix/735-generationid-abort-rollback
Open

[#735] Do not roll back a concurrently adopted generation ID on aborted handshake#736
vharseko wants to merge 2 commits into
OpenIdentityPlatform:masterfrom
vharseko:fix/735-generationid-abort-rollback

Conversation

@vharseko

Copy link
Copy Markdown
Member

Fixes #735.

Root cause

ServerHandler.abortStart blindly restored the domain generation ID to a snapshot taken (without the domain lock) before the handshake started. An outgoing RS handshake that lost a simultaneous cross-connect race could thereby wipe a generation ID the domain had legitimately adopted from peer gossip in the meantime — and clear the changelog DB — leaving one RS at -1 forever. This is the remaining GenerationIdTest.testMultiRS flake after #725 (the re-advertise fix couldn't help: the adopted value was received and then destroyed locally, and no further advertisement ever comes).

Changes

  • Arm the rollback only where the handler actually changes the generation id, via a new compare-and-set ReplicationServerDomain.changeGenerationIdIfUnchanged(expected, new) against the value the arming decision was based on (localGenerationId). This also fixes the stale-snapshot stomp on the success path (a genId concurrently adopted through the lock-free put()setGenerationIdIfUnset path can no longer be overwritten) and makes double-arming within one handshake harmless (previously connect() armed twice — OpenDJ-121 branch + checkGenerationId — and the second arm turned the abort rollback into a no-op).
  • Roll back before releasing the domain lock, via rollbackGenerationIdIfUnchanged: a handshake queued on the lock can no longer read, advertise or arm on the doomed value; the rollback is skipped once the generation id has been saved to the changelog or while DSs are connected (the same invariant resetGenerationIdIfPossible enforces), so it can never clear a changelog holding real changes.
  • Re-advertise every generation id transition, including resets to -1, so peers that recorded the previous value converge instead of diverging until the next unrelated topology event.
  • Hardening: wire generation ids below -1 are rejected at arming (a malformed peer could otherwise install -100 and collide with the not-armed sentinel, silently disabling the next handler's rollback); setGenerationIdIfUnset is now synchronized on generationIDLock (was a bare check-then-write reachable without the domain lock).

Test

HandshakeAbortGenerationIdTest — a deterministic reproducer with no timing assumptions: the test plays the remote peer itself, so it fully controls the interleaving (the RS snapshots the genId before sending its start message; the test adopts a genId while the RS is blocked awaiting the answer; only then rejects with StopMsg). The oracle is the generation id advertised on the second connection attempt, which the same connector thread sends strictly after the abort completed (happens-before, no sleeps). The test fails on the old code with expected [4801] but found [-1] and passes with the fix. A second method covers the new CAS/guard primitives, including the saved-generation-id guard.

Verification

  • HandshakeAbortGenerationIdTest 2/2 (and confirmed to fail without the product fix)
  • GenerationIdTest 4/4 (including the previously flaky testMultiRS and the two dependent tests), repeated runs
  • InitOnLineTest 10/10, ProtocolWindowTest, ReSyncTest, TopologyViewTest — all green

Root-cause trace analysis of the failing CI run is in #735.

…neration ID on aborted handshake

ServerHandler.abortStart blindly restored the domain generation ID to a
snapshot taken (without the domain lock) before the handshake started. An
outgoing RS handshake losing a simultaneous cross-connect race could thereby
wipe a generation ID the domain had legitimately adopted from peer gossip in
the meantime - and clear the changelog DB - leaving one RS at -1 forever
(GenerationIdTest.testMultiRS flake).

- arm the rollback only where the handler actually changes the generation id,
  via a compare-and-set (changeGenerationIdIfUnchanged) against the value the
  decision was based on: fixes the stale-snapshot stomp on the success path
  and makes double-arming within one handshake harmless
- roll back before releasing the domain lock, via rollbackGenerationIdIfUnchanged,
  and skip the rollback once the generation id was saved or DSs are connected
- re-advertise every generation id transition, including resets to -1, so
  peers converge instead of diverging until the next topology event
- reject wire generation ids below -1 (sentinel collision hardening) and
  synchronize setGenerationIdIfUnset on generationIDLock
- add HandshakeAbortGenerationIdTest: a deterministic reproducer driving the
  handshake from a scripted fake peer (fails on the old code), plus unit
  coverage of the new CAS/guard primitives
@vharseko vharseko added bug replication concurrency Thread-safety / race-condition bugs tests Test suites: fixing, enabling, un-disabling labels Jul 14, 2026
@vharseko vharseko requested a review from maximthomas July 14, 2026 06:38
initializeExportMultiSS flaked on a loaded CI runner: the broker session is
opened with soTimeout=1000ms, but waitForInitializeTargetMsg needs a full
round trip through a replication server mesh formed in the same second
(request routing across two RSs, InitializeTask scheduling on the DS, and
the InitializeTargetMsg routed back). One second of silence is not enough
headroom under CI load; raise all broker timeouts in this class to 10s.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug concurrency Thread-safety / race-condition bugs replication tests Test suites: fixing, enabling, un-disabling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replication: ServerHandler.abortStart rolls back a concurrently-adopted generation ID on cross-connect abort (GenerationIdTest.testMultiRS flake)

1 participant