Flume部署及应用实例
Flume部署及应用实例
版本下载
安装
- 将下载下来的apache-flume-1.6.0-bin.tar.gz包解压（tar -zxvf apache-flume-1.6.0-bin.tar.gz）到 /opt/目录下面
/etc/profile文件下添加
export FLUME_HOME=/opt/apache-flume-1.6.0-bin export PATH=$PATH:$FLUME_HOME/bin:$FLUME_HOME/conf验证是否成功
[root@node68 apache-flume-1.6.0-bin]# bin/flume-ng version Flume 1.6.0 Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080 Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015 From source with checksum b29e416802ce9ece3269d34233baf43f [root@node68 apache-flume-1.6.0-bin]#注意：出现上面的信息，表示安装成功了.
flume案例
- 案例一：Syslog
在conf目录下面添加 syslog.conf文件
[root@node68 conf]# pwd
/opt/apache-flume-1.6.0-bin/conf
[root@node68 conf]# ll
total 20
-rw-r–r–. 1 cfca games 1661 May 9 2015 flume-conf.properties.template
-rw-r–r–. 1 cfca games 1110 May 9 2015 flume-env.ps1.template
-rw-r–r–. 1 cfca games 1214 Aug 19 15:05 flume-env.sh.template
-rw-r–r–. 1 cfca games 3107 May 9 2015 log4j.properties
-rw-r–r–. 1 root root 1689 Aug 19 17:17 syslog.conf
[root@node68 conf]#
文件内容为：
# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'
agent1.sources = r1
agent1.channels = c1
agent1.sinks = k1
# For each one of the sources, the type is defined
agent1.sources.r1.type = syslogtcp
# The channel can be defined as follows.
agent1.sources.r1.channels = c1
agent1.sources.r1.port = 5410
agent1.sources.r1.host = 192.168.1.68
# Each sink's type must be defined
agent1.sinks.k1.type = logger
#Specify the channel the sink should use
agent1.sinks.k1.channel = c1
# Each channel's type is defined.
agent1.channels.c1.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent1.channels.c1.capacity = 1000
agent1.channels.c1.transactionCapacity = 100
启动flume agent agent1
[root@node68 apache-flume-1.6.0-bin]# bin/flume-ng agent -c conf -f conf/syslog.conf -n agent1 -Dflume.root.logger=INFO,console
测试产生syslog
[root@node68 apache-flume-1.6.0-bin]# echo "hello world syslog" | nc 192.168.1.68 5140控制台信息显示
2016-08-19 17:41:01,895 (New I/O worker #1) [WARN - org.apache.flume.source.SyslogUtils.buildEvent(SyslogUtils.java:316)] Event created from Invalid Syslog data. 2016-08-19 17:41:01,903 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{Severity=0, Facility=0, flume.syslog.status=Invalid} body: 68 65 6C 6C 6F 20 69 64 6F 61 6C 6C 2E 6F 72 67 hello world }
- 案例二：Spool
Spool监测配置的目录下新增的文件，并将文件中的数据读取出来。需要注意两点:1.拷贝到spool目录下的文件不可以再打开编辑。2.spool目录下不可包含相应的子目录
- 创建agent配置文件（spool.conf）
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.channels = c1
a1.sources.r1.spoolDir = /opt/apache-flume-1.6.0-bin/logs
a1.sources.r1.fileHeader = true
# # Describe the sink
a1.sinks.k1.type = logger
# # Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# # Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
启动flume agent a1
[root@node68 apache-flume-1.6.0-bin]# flume-ng agent -c conf -f conf/spool.conf -n a1 -Dflume.root.logger=INFO,console
追加文件到 到 /opt/apache-flume-1.6.0-bin/logs 目录
[root@node68 conf]# echo "spool test1" > /opt/apache-flume-1.6.0-bin/logs/spool_text3.log
控制台可看到日志
2016-08-30 08:50:55,628 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:258)] Last read took us just up to a file boundary. Rolling to the next file, if there is one. 2016-08-30 08:50:55,637 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:348)] Preparing to move file /opt/apache-flume-1.6.0-bin/logs/spool_text3.log to /opt/apache-flume-1.6.0-bin/logs/spool_text3.log.COMPLETED 2016-08-30 08:50:57,106 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{file=/opt/apache-flume-1.6.0-bin/logs/spool_text3.log} body: 73 70 6F 6F 6C 20 74 65 73 74 31 spool test1 }
