[#735] Do not roll back a concurrently adopted generation ID on aborted handshake#736
Open
vharseko wants to merge 2 commits into
Open
Conversation
…neration ID on aborted handshake ServerHandler.abortStart blindly restored the domain generation ID to a snapshot taken (without the domain lock) before the handshake started. An outgoing RS handshake losing a simultaneous cross-connect race could thereby wipe a generation ID the domain had legitimately adopted from peer gossip in the meantime - and clear the changelog DB - leaving one RS at -1 forever (GenerationIdTest.testMultiRS flake). - arm the rollback only where the handler actually changes the generation id, via a compare-and-set (changeGenerationIdIfUnchanged) against the value the decision was based on: fixes the stale-snapshot stomp on the success path and makes double-arming within one handshake harmless - roll back before releasing the domain lock, via rollbackGenerationIdIfUnchanged, and skip the rollback once the generation id was saved or DSs are connected - re-advertise every generation id transition, including resets to -1, so peers converge instead of diverging until the next topology event - reject wire generation ids below -1 (sentinel collision hardening) and synchronize setGenerationIdIfUnset on generationIDLock - add HandshakeAbortGenerationIdTest: a deterministic reproducer driving the handshake from a scripted fake peer (fails on the old code), plus unit coverage of the new CAS/guard primitives
initializeExportMultiSS flaked on a loaded CI runner: the broker session is opened with soTimeout=1000ms, but waitForInitializeTargetMsg needs a full round trip through a replication server mesh formed in the same second (request routing across two RSs, InitializeTask scheduling on the DS, and the InitializeTargetMsg routed back). One second of silence is not enough headroom under CI load; raise all broker timeouts in this class to 10s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #735.
Root cause
ServerHandler.abortStartblindly restored the domain generation ID to a snapshot taken (without the domain lock) before the handshake started. An outgoing RS handshake that lost a simultaneous cross-connect race could thereby wipe a generation ID the domain had legitimately adopted from peer gossip in the meantime — and clear the changelog DB — leaving one RS at-1forever. This is the remainingGenerationIdTest.testMultiRSflake after #725 (the re-advertise fix couldn't help: the adopted value was received and then destroyed locally, and no further advertisement ever comes).Changes
ReplicationServerDomain.changeGenerationIdIfUnchanged(expected, new)against the value the arming decision was based on (localGenerationId). This also fixes the stale-snapshot stomp on the success path (a genId concurrently adopted through the lock-freeput()→setGenerationIdIfUnsetpath can no longer be overwritten) and makes double-arming within one handshake harmless (previouslyconnect()armed twice — OpenDJ-121 branch +checkGenerationId— and the second arm turned the abort rollback into a no-op).rollbackGenerationIdIfUnchanged: a handshake queued on the lock can no longer read, advertise or arm on the doomed value; the rollback is skipped once the generation id has been saved to the changelog or while DSs are connected (the same invariantresetGenerationIdIfPossibleenforces), so it can never clear a changelog holding real changes.-1, so peers that recorded the previous value converge instead of diverging until the next unrelated topology event.-1are rejected at arming (a malformed peer could otherwise install-100and collide with the not-armed sentinel, silently disabling the next handler's rollback);setGenerationIdIfUnsetis now synchronized ongenerationIDLock(was a bare check-then-write reachable without the domain lock).Test
HandshakeAbortGenerationIdTest— a deterministic reproducer with no timing assumptions: the test plays the remote peer itself, so it fully controls the interleaving (the RS snapshots the genId before sending its start message; the test adopts a genId while the RS is blocked awaiting the answer; only then rejects withStopMsg). The oracle is the generation id advertised on the second connection attempt, which the same connector thread sends strictly after the abort completed (happens-before, no sleeps). The test fails on the old code withexpected [4801] but found [-1]and passes with the fix. A second method covers the new CAS/guard primitives, including the saved-generation-id guard.Verification
HandshakeAbortGenerationIdTest2/2 (and confirmed to fail without the product fix)GenerationIdTest4/4 (including the previously flakytestMultiRSand the two dependent tests), repeated runsInitOnLineTest10/10,ProtocolWindowTest,ReSyncTest,TopologyViewTest— all greenRoot-cause trace analysis of the failing CI run is in #735.