Skip to content

12.2.linux.kafka.md

Latest commit

 

History

History
247 lines (175 loc) · 5.5 KB

12.2.linux.kafka.md

File metadata and controls

247 lines (175 loc) · 5.5 KB

Kafka 安装使用教程

​

​

1 下载

官网: https://kafka.apache.org

下载: https://kafka.apache.org/downloads

​

2 安装

Linux/centOS 7 安装使用教程

下载:

wget https://downloads.apache.org/kafka/3.1.0/kafka_2.13-3.1.0.tgz

解压:

tar -zxvf kafka_2.13-3.1.0.tgz

进入目录:

cd kafka_2.13-3.1.0

修改 ZooKeeper 配置文件

vim config/zookeeper.properties

根据需要修改配置信息，也可以将 ZooKeeper 的配置文件复制到这里

修改 Kafka 配置文件

vim config/server.properties

常用配置属性:

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
advertised.listeners=PLAINTEXT://your.publicIP:9092
# A comma separated list of directories under which to store log files
log.dirs=/var/log/kafka/logs-1
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=127.0.0.1:2181

启动 ZooKeeper 并后台运行(如果使用 ZooKeeper 自带的脚本启动则略过这一步)

bin/zookeeper-server-start.sh config/zookeeper.properties &

启动 Kafka 并后台运行

bin/kafka-server-start.sh config/server.properties &

停止 Zookeeper 服务:

bin/zookeeper-server-stop.sh

停止 Kafka 服务:

bin/kafka-server-stop.sh

​

3 使用

3.1 依赖

依赖: Kafka 必须依赖 JDK,ZooKeeper

3.2 设置密码

修改 kafka 配置文件

config/server.properties

listeners=SASL_PLAINTEXT://:9092
advertised.listeners=SASL_PLAINTEXT://public.ip:9092
# security
# 认证协议
security.inter.broker.protocol=SASL_PLAINTEXT
# SASL 机制
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN

创建密码配置文件

kafka-server-jaas.conf

KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
        username="serverUser"
        password="serverPass@3124"
        user_serverUser="serverPass@3124";
};

注意事项:

用户名为: serverUser

密码: serverPass@3124

最后一行格式: user_用户名=密码

严格按照以上格式书写，不能有多余的空行和标点

修改 kafka 启动脚本

bin/kafka-server-start.sh

添加代码

if [ "x$KAFKA_OPTS"  ]; then
    export KAFKA_OPTS="-Djava.security.auth.login.config=/usr/kafka_2.13-3.1.0/config/kafka_server_jaas.conf"
fi

根据 kafka 实际安装路径进行修改

命令必须添加在以下代码之前

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

springboot application 配置

## spring
spring:
  application:
    name: demo-kafka
  kafka:
    bootstrap-servers: public.ip:9092
    consumer:
      group-id: 1
      enable-auto-commit: true
      auto-commit-interval: 100ms
      properties:
        session.timeout.ms: 15000
        sasl.mechanism: PLAIN
        security.protocol: SASL_PLAINTEXT
        sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username='serverUser' password='serverPass@3124';
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      auto-offset-reset: earliest
    producer:
      retries: 0
      batch-size: 16384
      buffer-memory: 33554432
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      properties:
        sasl.mechanism: PLAIN
        security.protocol: SASL_PLAINTEXT
        sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username='serverUser' password='serverPass@3124';

推荐参考资料:

为Kafka 添加用户名 密码权限，即SASL/PLAIN

kafka密码SASL认证配置及SpringBoot配置加密后的Kafka参数

3.3 修改端口

Kafka 默认端口为 9092

修改 kafka 启动配置文件

config/server.properties

listeners=SASL_PLAINTEXT://:9099
advertised.listeners=SASL_PLAINTEXT://public.ip:9099

单节点部署修改以下配置文件

config/connect-standalone.properties

bootstrap.servers=localhost:9099

集群部署修改以下配置文件

config/connect-distributed.properties

bootstrap.servers=localhost:9099

推荐参考资料:

kafka端口号修改