Flume 的負載均衡

負載均衡是用于解決一臺機?(一個進程)無法解決所有請求而產生的一種算法。Load balancing SinkProcessor 能夠實現 load balance 功能，如下圖Agent1 是一個路由節點，負責將 Channel 暫存的Event 均衡到對應的多個 Sink組件上，而每個 Sink 組件分別連接到一個獨立的 Agent 上，示例配置，如下所示：

在此處我們通過三臺機?來進行模擬flume的負載均衡

三臺機?規劃如下：

node01：采集數據，發送到node02和node03機?上去

node02：接收node01的部分數據

node03：接收node01的部分數據

第一步：開發node01服務?的flume配置

node01服務?配置：

cd /export/servers/apache-flume-1.8.0-bin/conf vim load_banlancer_client.conf # agent name a1.channels = c1 a1.sources = r1 a1.sinks = k1 k2# set gruop a1.sinkgroups = g1# set channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.sources.r1.channels = c1 a1.sources.r1.type = exec a1.sources.r1.command = tail -F /export/servers/taillogs/access_log# set sink1 a1.sinks.k1.channel = c1 a1.sinks.k1.type = avro a1.sinks.k1.hostname = node02 a1.sinks.k1.port = 52020# set sink2 a1.sinks.k2.channel = c1 a1.sinks.k2.type = avro a1.sinks.k2.hostname = node03 a1.sinks.k2.port = 52020# set sink group a1.sinkgroups.g1.sinks = k1 k2# set failover a1.sinkgroups.g1.processor.type = load_balance a1.sinkgroups.g1.processor.backoff = true a1.sinkgroups.g1.processor.selector = round_robin a1.sinkgroups.g1.processor.selector.maxTimeOut=10000

第二步：開發node02服務?的flume配置

cd /export/servers/apache-flume-1.8.0-bin/conf vim load_banlancer_server.conf# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1# Describe/configure the source a1.sources.r1.type = avro a1.sources.r1.channels = c1 a1.sources.r1.bind = node02 a1.sources.r1.port = 52020# Describe the sink a1.sinks.k1.type = logger# Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1

第三步：開發node03服務?flume配置

node03服務?配置

cd /export/servers/apache-flume-1.8.0-bin/conf vim load_banlancer_server.conf # Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1# Describe/configure the source a1.sources.r1.type = avro a1.sources.r1.channels = c1 a1.sources.r1.bind = node03 a1.sources.r1.port = 52020# Describe the sink a1.sinks.k1.type = logger# Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1

第四步：準備啟動flume服務

啟動node03的flume服務

cd /export/servers/apache-flume-1.8.0-bin bin/flume-ng agent -n a1 -c conf -f conf/load_banlancer_server.conf -Dflume.root.logger

啟動node02的flume服務

cd /export/servers/apache-flume-1.8.0-bin bin/flume-ng agent -n a1 -c conf -f conf/load_banlancer_server.conf -Dflume.root.logger

啟動node01的flume服務

cd /export/servers/apache-flume-1.8.0-bin bin/flume-ng agent -n a1 -c conf -f conf/load_banlancer_client.conf -Dflume.root.logger

第五步：node01服務?運行腳本產生數據

cd /export/servers/shells sh tail-file.sh

?

總結

以上是生活随笔為你收集整理的flume高可用-balance-配置文件编写的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。