签到成功

知道了

CNDBA社区CNDBA社区

CDH 5.16 安装 Kafka 集群 操作手册

2019-05-27 16:47 2053 0 原创 Kafka
作者: Dave

1下载Kafaka的parcel包

Kafka兼容性列表可以参考Cloudera的官方手册:
https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_kafka

https://www.cndba.cn/dave/article/3415

这里从官方下载parcel包:
http://archive.cloudera.com/kafka/parcels/4.0.0.1/

https://www.cndba.cn/dave/article/3415

根据操作系统的对应版本下载:

https://www.cndba.cn/dave/article/3415

[dave@www.cndba.cn ~]# uname -a
Linux hadoop-cm 2.6.32-696.el6.x86_64 #1 SMP Tue Feb 21 00:53:17 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
[dave@www.cndba.cn ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.9 (Santiago)
[dave@www.cndba.cn ~]#

我们这里下载:

https://www.cndba.cn/dave/article/3415

http://archive.cloudera.com/kafka/parcels/4.0.0.1/KAFKA-4.0.0-1.4.0.0.p0.1-el6.parcel
http://archive.cloudera.com/kafka/parcels/4.0.0.1/KAFKA-4.0.0-1.4.0.0.p0.1-el6.parcel.sha1
http://archive.cloudera.com/kafka/parcels/4.0.0.1/manifest.json

下载完成后将parcel.sha1文件重命名为parcel.sha。

将下载的3个文件上传到Cloudear的目录:

https://www.cndba.cn/dave/article/3415
https://www.cndba.cn/dave/article/3415

[dave@www.cndba.cn parcel-repo]# pwd
/opt/cloudera/parcel-repo
[dave@www.cndba.cn parcel-repo]# ll
total 1817740
-rw-r--r--. 1 root root 1861213140 Apr 30  2019 CDH-5.16.1-1.cdh5.16.1.p0.3-el6.parcel
-rw-r--r--. 1 root root         41 Apr 30  2019 CDH-5.16.1-1.cdh5.16.1.p0.3-el6.parcel.sha
-rw-r-----. 1 root root      71166 Apr 30  2019 CDH-5.16.1-1.cdh5.16.1.p0.3-el6.parcel.torrent
-rw-r--r--. 1 root root      66538 Apr 30  2019 manifest.json
[dave@www.cndba.cn parcel-repo]#

下载KAFKA-1.2.0.jar文件:

http://archive.cloudera.com/csds/kafka/
将文件放到CM节点的/opt/cloudera/csd目录下。

2分配并激活kafka

长传parcel包之后,正常还是不能显示,点击检查新Parcel,或者重启CM后,正确识别出Kafka的parcel包,点击分配并激活。


3安装Kafka集群

CM—> 添加服务 —> Kafka.

填写Destination Brokers List
填写添加了Kafka MirrorMaker的节点的ip+端口号,比如:hadoop-tool:9092

https://www.cndba.cn/dave/article/3415

填写Source Brokers List: 即Kafka Broker所在节点的ip+端口号,我这里填写的信息如下:https://www.cndba.cn/dave/article/3415

hadoop-work1:9092,hadoop-work1:9092,hadoop-work1:9092

启动的时候会报如下错误:

https://www.cndba.cn/dave/article/3415

Date: Fri Nov 24 18:21:40 CST 2028
Host: hadoop-work3
Pwd: /opt/cm-5.16.1/run/cloudera-scm-agent/process/514-kafka-KAFKA_BROKER
CONF_DIR: /opt/cm-5.16.1/run/cloudera-scm-agent/process/514-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/lib/kafka
Zookeeper Quorum: hadoop-master1:2181,hadoop-master2:2181,hadoop-master3:2181
Zookeeper Chroot: 
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 50
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL: 
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 2.1.0-kafka4.0.0
Sentry version found: 1.5.1-cdh5.16.1
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is hadoop-master1:2181,hadoop-master2:2181,hadoop-master3:2181
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://hadoop-work3:9092,
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/kafka_kafka-KAFKA_BROKER-92c62d373d0ea0ddd4d7f857102caca0_pid21306.hprof ...
Heap dump file created [13863418 bytes in 0.153 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="/opt/cm-5.16.1/lib64/cmf/service/common/killparent.sh"
#   Executing /bin/sh -c "/opt/cm-5.16.1/lib64/cmf/service/common/killparent.sh"...

返回kafka配置界面,修改broker_max_heap_size和mirror_maker_max_heap_size参数到300M,然后启动正常。默认值50M不满足需求。

https://www.cndba.cn/dave/article/3415

但MirrorMaker 依旧启动失败,查看角色日志:

Starting mirror maker
whitelist must be specified

修改whitelist参数,添加如下内容:

hadoop-work1:9092,hadoop-work1:9092,hadoop-work1:9092

启动正常:

版权声明:本文为博主原创文章,未经博主允许不得转载。

用户评论
* 以下用户言论只代表其个人观点,不代表CNDBA社区的观点或立场
Dave

Dave

关注

人的一生应该是这样度过的:当他回首往事的时候,他不会因为虚度年华而悔恨,也不会因为碌碌无为而羞耻;这样,在临死的时候,他就能够说:“我的整个生命和全部精力,都已经献给世界上最壮丽的事业....."

  • 1750
    原创
  • 2
    翻译
  • 389
    转载
  • 97
    评论
  • 访问:2256174次
  • 积分:2618
  • 等级:核心会员
  • 排名:第1名
精华文章
    最新问题
    查看更多+
    热门文章
      热门用户
      推荐用户
        Copyright © 2016 All Rights Reserved. Powered by ZhiXinTech · 皖ICP备19020168号·

        QQ交流群