Skip to content

add: multi Topic support#10

Open
GefMar wants to merge 1 commit intomainfrom
MultyTopicBrocker
Open

add: multi Topic support#10
GefMar wants to merge 1 commit intomainfrom
MultyTopicBrocker

Conversation

@GefMar
Copy link
Copy Markdown

@GefMar GefMar commented Apr 26, 2026

Summary

Adds multi-topic support to AioKafkaBroker while keeping the existing single-topic behavior fully backward compatible.

Why

Some projects need different task groups to use different Kafka topics, while still running through the same taskiq broker and worker setup. For example, regular tasks, email tasks, reports, or priority-specific workloads may need separate Kafka topics for routing, isolation, monitoring, or operational control.

Previously, AioKafkaBroker always sent every task to one broker-level Kafka topic. This change allows each task to define its own default topic, and also allows overriding the topic for a single kiq call.

What Changed

  • Added TopicConfig and Topic helper structures.
  • Extended AioKafkaBroker with multi-topic support:
    • kafka_topic remains the default topic.
    • kafka_topics can be used to configure additional topics for the worker to listen to.
    • @broker.task(topic="...") sets a default Kafka topic for that task.
    • task.kicker().with_topic("...").kiq(...) overrides the topic for a single task kick.
  • Updated startup so worker consumers subscribe to all configured topics.
  • Updated shutdown topic cleanup to handle all configured topics when delete_topic_on_shutdown=True.
  • Preserved backward compatibility for existing AioKafkaBroker usage.
  • Added tests for:
    • task-level default topic routing;
    • per-kick topic override;
    • fallback to broker default topic;
    • consumer subscription to all topics.
  • Updated README with multi-topic usage examples.

@GefMar GefMar requested a review from danfimov April 26, 2026 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant