Node01 安裝和配置

將node03機?上面的flume安裝包以及文件生產的兩個目錄拷貝到node01機?上面去

node03機?執行以下命令

cd /export/servers scp -r apache-flume-1.8.0-bin/ node01:$PWD scp -r shells/ taillogs/ node01:$PWD

node01機?配置agent的配置文件

cd /export/servers/apache-flume-1.8.0-bin/conf vim agent.conf #agent1 name agent1.channels = c1 agent1.sources = r1 agent1.sinks = k1 k2 # ##set gruop agent1.sinkgroups = g1 # agent1.sources.r1.channels = c1 agent1.sources.r1.type = exec agent1.sources.r1.command = tail -F /export/servers/taillogs/access_log # ##set channel agent1.channels.c1.type = memory agent1.channels.c1.capacity = 1000 agent1.channels.c1.transactionCapacity = 100 # ## set sink1 agent1.sinks.k1.channel = c1 agent1.sinks.k1.type = avro agent1.sinks.k1.hostname = node02 agent1.sinks.k1.port = 52020 # ## set sink2 agent1.sinks.k2.channel = c1 agent1.sinks.k2.type = avro agent1.sinks.k2.hostname = node03 agent1.sinks.k2.port = 52020 # ##set sink group agent1.sinkgroups.g1.sinks = k1 k2 # ##set failover agent1.sinkgroups.g1.processor.type = failover agent1.sinkgroups.g1.processor.priority.k1 = 10 agent1.sinkgroups.g1.processor.priority.k2 = 1 agent1.sinkgroups.g1.processor.maxpenalty = 10000

Node02 與 Node03 配置 FlumeCollection

node02機?修改配置文件

cd /export/servers/apache-flume-1.8.0-bin/conf vim collector.conf #set Agent name a1.sources = r1 a1.channels = c1 a1.sinks = k1 # ##set channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # ## other node,nna to nns a1.sources.r1.type = avro a1.sources.r1.bind = node02 a1.sources.r1.port = 52020 a1.sources.r1.channels = c1 # ##set sink to hdfs a1.sinks.k1.type=hdfs a1.sinks.k1.hdfs.path= hdfs://node01:8020/flume/failover/ a1.sinks.k1.hdfs.fileType=DataStream a1.sinks.k1.hdfs.writeFormat=TEXT a1.sinks.k1.hdfs.rollInterval=10 a1.sinks.k1.channel=c1 a1.sinks.k1.hdfs.filePrefix=%Y-%m-%d #

node03機?修改配置文件

cd /export/servers/apache-flume-1.8.0-bin/conf vim collector.conf #set Agent name a1.sources = r1 a1.channels = c1 a1.sinks = k1 # ##set channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # ## other node,nna to nns a1.sources.r1.type = avro a1.sources.r1.bind = node03 a1.sources.r1.port = 52020 a1.sources.r1.channels = c1 # ##set sink to hdfs a1.sinks.k1.type=hdfs a1.sinks.k1.hdfs.path= hdfs://node01:8020/flume/failover/ a1.sinks.k1.hdfs.fileType=DataStream a1.sinks.k1.hdfs.writeFormat=TEXT a1.sinks.k1.hdfs.rollInterval=10 a1.sinks.k1.channel=c1 a1.sinks.k1.hdfs.filePrefix=%Y-%m-%d

順序啟動

node03機?上面啟動flume

cd /export/servers/apache-flume-1.8.0-bin bin/flume-ng agent -n a1 -c conf -f conf/collector.conf -Dflume.root.logger=DEBUG,conso

node02機?上面啟動flume

cd /export/servers/apache-flume-1.8.0-bin bin/flume-ng agent -n a1 -c conf -f conf/collector.conf -Dflume.root.logger=DEBUG,conso

node01機?上面啟動flume

cd /export/servers/apache-flume-1.8.0-bin bin/flume-ng agent -n agent1 -c conf -f conf/agent.conf -Dflume.root.logger=DEBUG,conso

node01機?啟動文件產生腳本

cd /export/servers/shells sh tail-file.sh

?

總結

以上是生活随笔為你收集整理的flume高可用-failover-配置文件编写的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。