签到成功

知道了

CNDBA社区CNDBA社区

Flume1.6安装部署测试

2019-05-20 17:44 3907 0 原创 Flume
作者： lirui
本文链接：http://www.cndba.cn/lirui/article/3412

1.环境

CentOS 6.8
flume 1.6.0

2.下载并安装

下载地址：http://flume.apache.org/download.html

http://www.cndba.cn/lirui/article/3412

也可以通过wget直接下载:

wget http://mirrors.hust.edu.cn/apache/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz

3.安装

我下载到的目录是/data/software，安装到/opt/目录下

tar -zxvf apache-flume-1.6.0-bin.tar.gz -C /opt

4.将apache-flume-1.6.0-bin改名为apache-flume-1.6.0

mv apache-flume-1.8.0-bin/ apache-flume-1.8.0

5.添加flume环境变量

vi /etc/profile

添加上

export JAVA_HOME=/opt/jdk1.8
export FLUME_HOME=/opt/apache-flume-1.6.0
export FLUME_CONF_DIR=$FLUME_HOME/conf
export PATH=$JAVA_HOME/bin:$FLUME_HOME/bin:$PATH

6.修改flume-env.sh

cd /opt/apache-flume-1.6.0/conf
# 拷贝出模板
cp flume-env.sh.template flume-env.sh
# 修改配置文件
vi flume-env.sh

添加上http://www.cndba.cn/lirui/article/3412

export JAVA_HOME=/opt/jdk1.8
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

7.修改flume-conf

cp flume-conf.properties.template flume-conf.properties
vi flume-conf.properties

8.添加agent、source、channel、sink的相关信息

a1.sources=r1
a1.channels=c1
a1.sinks=k1
#Describe/configure the source
a1.sources.r1.type=netcat
a1.sources.r1.bind=192.168.30.125
a1.sources.r1.port=44444
a1.sources.r1.channels=c1
#Describe the sink
a1.sinks.k1.type=logger
a1.sinks.k1.channel=c1
#Describe the channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transcationCapacity=100

9.启动

注：位于flume的安装目录启动http://www.cndba.cn/lirui/article/3412

cd /opt/apache-flume-1.6.0
# 启动命令
./bin/flume-ng agent --conf ./conf --conf-file ./conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console

输出：

Info: Sourcing environment configuration script /opt/apache-flume-1.6.0/conf/flume-env.sh
Info: Including Hive libraries found via () for Hive access
+ exec /opt/jdk1.8/bin/java -Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dflume.root.logger=INFO,console -cp '/opt/apache-flume-1.6.0/conf:/opt/apache-flume-1.6.0/lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file ./conf/flume-conf.properties --name a1
2019-05-20 13:22:38,428 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2019-05-20 13:22:38,444 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:./conf/flume-conf.properties
2019-05-20 13:22:38,464 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:931)] Added sinks: k1 Agent: a1
2019-05-20 13:22:38,464 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:k1
2019-05-20 13:22:38,464 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:k1
2019-05-20 13:22:38,501 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:141)] Post-validation flume configuration contains configuration for agents: [a1]
2019-05-20 13:22:38,502 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:145)] Creating channels
2019-05-20 13:22:38,549 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel c1 type memory
2019-05-20 13:22:38,558 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:200)] Created channel c1
2019-05-20 13:22:38,559 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source r1, type netcat
2019-05-20 13:22:38,579 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: k1, type: logger
2019-05-20 13:22:38,587 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:114)] Channel c1 connected to [r1, k1]
2019-05-20 13:22:38,597 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@43bbd949 counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
2019-05-20 13:22:38,619 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel c1
2019-05-20 13:22:38,627 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2019-05-20 13:22:38,627 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: CHANNEL, name: c1 started
2019-05-20 13:22:38,629 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink k1
2019-05-20 13:22:38,648 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source r1
2019-05-20 13:22:38,650 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:150)] Source starting
2019-05-20 13:22:38,665 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:164)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/192.168.30.125:44444]

flume启动成功http://www.cndba.cn/lirui/article/3412

说明：启动命令格式

./bin/flume-ng agent --conf <conf路径> --conf-file <flume-conf.properties路径> --name <agent名称> -Dflume.root.logger=INFO,console

注意：-Dflume.root.logger=INFO,console 只有一个短横线

9.测试

在另外一台机器上192.168.20.165上telnet开启flume的这台机器http://www.cndba.cn/lirui/article/3412http://www.cndba.cn/lirui/article/3412http://www.cndba.cn/lirui/article/3412

192.168.20.165 输入hello world 

[root@toolnode2 ~]# telnet 192.168.30.125 44444
Trying 192.168.30.125...
Connected to 192.168.30.125.
Escape character is '^]'.
hello world
OK

控制台输出：

http://www.cndba.cn/lirui/article/3412 

2019-05-20 13:22:38,665 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:164)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/192.168.30.125:44444]
2019-05-20 13:28:13,067 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 0D             hello world. }

192.168.20.165 输入hello flumehttp://www.cndba.cn/lirui/article/3412

telnet 192.168.30.125 44444
Trying 192.168.30.125...
Connected to 192.168.30.125.
Escape character is '^]'.
hello world
OK
hello flume
OK

控制台输出：

2019-05-20 13:31:15,626 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 65 6C 6C 6F 20 66 6C 75 6D 65 0D             hello flume. }

说明：使用telnet命令，需先安装telnet

http://www.cndba.cn/lirui/article/3412 

yum install -y telnet

版权声明：本文为博主原创文章，未经博主允许不得转载。

用户评论
* 以下用户言论只代表其个人观点，不代表CNDBA社区的观点或立场
lirui

lirui

关注
  • 15
    原创
  • 0
    翻译
  • 0
    转载
  • 1
    评论
  • 访问：61824次
  • 积分：56
  • 等级：注册会员
  • 排名：第39名
精华文章

    达梦数据库运维实战

    Oracle 18c 必须掌握的新特性：管理与实战

    Oracle数据库问题解决方案和故障排除手册
    最新问题
    查看更多+
    热门文章
      热门用户
      推荐用户
        友情链接： CNDBA社区 | openGauss | openEuler | openAnolis | 广告联系: database@ustc.edu
        Copyright © 2016 All Rights Reserved. Powered by CNDBA · 皖ICP备2022006297号-1·

        QQ交流群

        注册联系QQ