CNDBA社区
1下载Kafaka的parcel包
Kafka兼容性列表可以参考Cloudera的官方手册：
https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_kafka
这里从官方下载parcel包：
http://archive.cloudera.com/kafka/parcels/4.0.0.1/
根据操作系统的对应版本下载：
[dave@www.cndba.cn ~]# uname -a
Linux hadoop-cm 2.6.32-696.el6.x86_64 #1 SMP Tue Feb 21 00:53:17 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
[dave@www.cndba.cn ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.9 (Santiago)
[dave@www.cndba.cn ~]#
我们这里下载：
http://archive.cloudera.com/kafka/parcels/4.0.0.1/KAFKA-4.0.0-1.4.0.0.p0.1-el6.parcel
http://archive.cloudera.com/kafka/parcels/4.0.0.1/KAFKA-4.0.0-1.4.0.0.p0.1-el6.parcel.sha1
http://archive.cloudera.com/kafka/parcels/4.0.0.1/manifest.json
下载完成后将parcel.sha1文件重命名为parcel.sha。
将下载的3个文件上传到Cloudear的目录：
[dave@www.cndba.cn parcel-repo]# pwd
/opt/cloudera/parcel-repo
[dave@www.cndba.cn parcel-repo]# ll
total 1817740
-rw-r--r--. 1 root root 1861213140 Apr 30 2019 CDH-5.16.1-1.cdh5.16.1.p0.3-el6.parcel
-rw-r--r--. 1 root root 41 Apr 30 2019 CDH-5.16.1-1.cdh5.16.1.p0.3-el6.parcel.sha
-rw-r-----. 1 root root 71166 Apr 30 2019 CDH-5.16.1-1.cdh5.16.1.p0.3-el6.parcel.torrent
-rw-r--r--. 1 root root 66538 Apr 30 2019 manifest.json
[dave@www.cndba.cn parcel-repo]#
下载KAFKA-1.2.0.jar文件：
http://archive.cloudera.com/csds/kafka/
将文件放到CM节点的/opt/cloudera/csd目录下。
2分配并激活kafka
长传parcel包之后，正常还是不能显示，点击检查新Parcel，或者重启CM后，正确识别出Kafka的parcel包，点击分配并激活。
3安装Kafka集群
CM—> 添加服务 —> Kafka.
填写Destination Brokers List
填写添加了Kafka MirrorMaker的节点的ip+端口号，比如：hadoop-tool:9092
填写Source Brokers List: 即Kafka Broker所在节点的ip+端口号，我这里填写的信息如下：
hadoop-work1:9092,hadoop-work1:9092,hadoop-work1:9092
启动的时候会报如下错误：
Date: Fri Nov 24 18:21:40 CST 2028
Host: hadoop-work3
Pwd: /opt/cm-5.16.1/run/cloudera-scm-agent/process/514-kafka-KAFKA_BROKER
CONF_DIR: /opt/cm-5.16.1/run/cloudera-scm-agent/process/514-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/lib/kafka
Zookeeper Quorum: hadoop-master1:2181,hadoop-master2:2181,hadoop-master3:2181
Zookeeper Chroot:
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 50
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL:
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 2.1.0-kafka4.0.0
Sentry version found: 1.5.1-cdh5.16.1
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is hadoop-master1:2181,hadoop-master2:2181,hadoop-master3:2181
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://hadoop-work3:9092,
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/kafka_kafka-KAFKA_BROKER-92c62d373d0ea0ddd4d7f857102caca0_pid21306.hprof ...
Heap dump file created [13863418 bytes in 0.153 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="/opt/cm-5.16.1/lib64/cmf/service/common/killparent.sh"
# Executing /bin/sh -c "/opt/cm-5.16.1/lib64/cmf/service/common/killparent.sh"...
返回kafka配置界面，修改broker_max_heap_size和mirror_maker_max_heap_size参数到300M，然后启动正常。默认值50M不满足需求。
但MirrorMaker 依旧启动失败，查看角色日志：
Starting mirror maker
whitelist must be specified
修改whitelist参数，添加如下内容：
hadoop-work1:9092,hadoop-work1:9092,hadoop-work1:9092
启动正常：
版权声明：本文为博主原创文章，未经博主允许不得转载。
