Skip to content

[feat] PIP-468: Add scalable topic protocol commands and connection handling#25564

Merged
merlimat merged 1 commit intoapache:masterfrom
merlimat:st-protocol
Apr 24, 2026
Merged

[feat] PIP-468: Add scalable topic protocol commands and connection handling#25564
merlimat merged 1 commit intoapache:masterfrom
merlimat:st-protocol

Conversation

@merlimat
Copy link
Copy Markdown
Contributor

Stacked on top of #25559. This PR is opened as a draft and will be
rebased on master once #25559 lands. Reviewers: the new code in this
PR is the single commit titled "PIP-468: Add scalable topic protocol
commands and connection handling"; everything else comes from #25559.

Summary

Second PR in the PIP-468 series. Introduces the wire protocol for the
scalable-topic client/broker interaction:

  • New protocol commands on PulsarApi.proto:
    CommandScalableTopicLookup, CommandScalableTopicUpdate,
    CommandScalableTopicClose, CommandScalableTopicSubscribe,
    CommandScalableTopicSubscribeResponse,
    CommandScalableTopicAssignmentUpdate.
  • Commands factory methods for building each message (including the
    error variants for lookup/subscribe).
  • PulsarDecoder dispatch entries for all six new message types.
  • Broker-side DagWatchSession — resolves per-segment broker
    addresses, watches the metadata store for DAG changes, and pushes
    ScalableTopicUpdate frames on the wire. Tied to the transport
    connection — closed when the connection drops.
  • ServerCnx handlers for ScalableTopicLookup and
    ScalableTopicClose.
  • Client-side DagWatchSession interface (callback surface for the V5
    client that lands later).
  • ClientCnx handler for ScalableTopicUpdate that dispatches to the
    registered session.
  • ConsumerSession.sendAssignmentUpdate wired through to the new
    command sender (previously a placeholder no-op).

Tests

Three new unit-test classes:

  • CommandsScalableTopicTest (8 tests) — every new factory method
    roundtrips through the full wire frame; uses
    BaseCommand.materialize() so the buffer can be safely released
    before assertions.
  • ConsumerSessionTest (15 tests) — covers identity/equality,
    attach/markDisconnected, grace-timer lifecycle, all three
    sendAssignmentUpdate branches (connected, disconnected, null
    command sender), and the toProto static helper.
  • DagWatchSessionTest (12 tests) — session identity, idempotent
    close, start() failure on missing topic, metadata-store listener
    registration, notification filtering (wrong path / Deleted /
    post-close), pushUpdate serializing the correct DAG to the
    connection, and post-close/no-broker-map edge cases.

The heavy NamespaceService lookup path inside start() /
buildResponse is deferred to broker integration tests — mocking the
full namespace stack here is more noise than signal.

Test plan

  • ./gradlew :pulsar-common:test --tests "org.apache.pulsar.common.protocol.CommandsScalableTopicTest"
    — 8/8 pass.
  • ./gradlew :pulsar-broker:test --tests "org.apache.pulsar.broker.service.scalable.*"
    — 61/61 pass (5 test classes, including the two existing ones
    from the previous PR).
  • ./gradlew :pulsar-broker:checkstyleMain :pulsar-broker:checkstyleTest :pulsar-common:checkstyleMain :pulsar-common:checkstyleTest
    — all clean.

@merlimat merlimat added type/feature The PR added a new feature or issue requested a new feature area/broker labels Apr 22, 2026
@merlimat merlimat changed the title PIP-468: Add scalable topic protocol commands and connection handling [feat] PIP-468: Add scalable topic protocol commands and connection handling Apr 22, 2026
Define ScalableTopicLookup, ScalableTopicUpdate, and ScalableTopicClose
protocol commands in PulsarApi.proto. Add Commands factory methods for
building/parsing these messages, including error responses. Implement
ServerCnx handlers for lookup (with DagWatchSession that resolves segment
broker addresses and pushes DAG updates) and close commands. Add ClientCnx
handler for ScalableTopicUpdate with DagWatchSession lifecycle management
and connection-closed cleanup.
@merlimat merlimat marked this pull request as ready for review April 24, 2026 19:56
@merlimat merlimat requested a review from lhotari April 24, 2026 19:59
@merlimat merlimat added this to the 5.0.0 milestone Apr 24, 2026
Copy link
Copy Markdown
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one minor question

Comment thread pulsar-common/src/main/proto/PulsarApi.proto
@merlimat merlimat merged commit 597eb0d into apache:master Apr 24, 2026
43 checks passed
@lhotari lhotari modified the milestones: 5.0.0, 5.0.0-M1 Apr 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/broker type/feature The PR added a new feature or issue requested a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants