Fix intermittent GenerationIdTest.testMultiRS by re-advertising genId on change

[vharseko]

Problem

GenerationIdTest.testMultiRS intermittently fails on CI (e.g. the ubuntu-latest/17 leg of PR #720's run, unrelated to that PR) with:

SoftAssertionError: 1) [in replServer3] expected:<[48]L> but was:<[-1]L>
  at GenerationIdTest.assertGenIdEquals(GenerationIdTest.java:1043)
  at GenerationIdTest.waitForStableGenerationId(GenerationIdTest.java:1027)   // repeatUntilSuccess, 120s
  at GenerationIdTest.testMultiRS(GenerationIdTest.java:927)

48 = GenerationIdChecksum.EMPTY_BACKEND_GENERATION_ID; -1 = "generationId not set". One replication server (replServer3) never converges on the common genId within the 120s wait.

Root cause

The generationId gossip between replication servers is purely event-driven — it is carried in the topology messages sent on connect/disconnect/status events, and there is no periodic re-advertisement. ReplicationServerDomain.changeGenerationId() silently updated the field and relied on a separate topology broadcast triggered by the surrounding event (e.g. register(DataServerHandler) → enqueueTopoInfoToAllExcept).

If a peer RS misses or races that single broadcast — it finished joining the RS mesh in the narrow window where the origin RS had not yet adopted the new genId (so the RS↔RS handshake advertised -1), and the one DS-register broadcast did not latch on it — nothing ever re-sends the value, so the peer stays stuck on the stale genId indefinitely (within the test window). That is exactly the observed expected:<48> but was:<-1>.

Prior fixes (#615 domain-ready wait, #618 timeout 60→120s, #622 wait for getNumRSs()>=2) only narrowed the window; they cannot cover a missed broadcast because a missed broadcast is never retried.

Fix

Re-advertise the topology (sendTopoInfoToAll) on every real generationId transition in changeGenerationId, so the gossip is self-healing: any peer that missed the first broadcast gets the next one, and every connected peer converges on the new value.

Why it is safe

• Idempotent & cheap — sendTopoInfoToAll only enqueues topology messages onto pendingStatusMessages and wakes the StatusAnalyzer; delivery is async. genId changes are rare in steady state, so the extra traffic is negligible. The reset path (resetGenerationId) already re-broadcasts; a second idempotent enqueue is harmless.
• Lock-safe — changeGenerationId holds only generationIDLock; sendTopoInfoToAll acquires pendingStatusMessagesLock + the analyzer eventMonitor, neither of which is ever held while acquiring generationIDLock, so no new lock-ordering/deadlock risk. pendingStatusMessages is initialised at field declaration and statusAnalyzer is started in the constructor, both before any changeGenerationId call.
• No convergence storm — a peer only re-broadcasts when its own genId changes; once every RS holds the value no further changes occur, so the gossip terminates.
• The change only adds idempotent re-advertisements; it cannot make convergence worse, only help peers that missed a broadcast.

Tests

GenerationIdTest (failsafe) locally: Tests run: 4, Failures: 0, Errors: 0 — BUILD SUCCESS.

Being a timing flake, a green local run confirms no regression but cannot by itself prove the race is gone — the safety argument above is the primary evidence. A CI soak (re-running the job repeatedly) is recommended to build confidence.

[maximthomas]

Suggested change

	// The generationId gossip between replication servers is purely
	// event-driven: it is carried in the topology messages sent on
	// connect/disconnect/status events, and there is no periodic
	// re-advertisement. A peer RS that misses (or races) the single
	// topology broadcast following a generationId change would otherwise
	// stay stuck with a stale generationId (typically -1) indefinitely -
	// the intermittent GenerationIdTest.testMultiRS failure. Re-advertising
	// the topology on every real generationId transition makes the gossip
	// self-healing so every connected peer converges on the new value.
	sendTopoInfoToAll();
	// generationId gossip is purely event-driven: it only travels in the
	// topology messages sent on connect/disconnect/status events. Re-advertise
	// on every real transition so a peer that missed one converges on the next.
	if (generationId > 0)
	{
	sendTopoInfoToAll();
	}

Edit: less verbose comment message

[vharseko]

Applied verbatim in 17d2ae1. The generationId > 0 guard is safe for both -1 paths: mayResetGenerationId() is a local cleanup each RS performs independently, and the ResetGenerationIdMsg path already forwards the reset to every connected RS itself. GenerationIdTest 4/4 green locally with the guard.