Let's put together a few pieces we touched on earlier and see how one Flume agent can use another as its sink.
agent6.conf
:agent6.sources = avrosource agent6.sinks = avrosink agent6.channels = memorychannel agent6.sources.avrosource.type = avro agent6.sources.avrosource.bind = localhost agent6.sources.avrosource.port = 2000 agent6.sources.avrosource.threads = 5 agent6.sinks.avrosink.type = avro agent6.sinks.avrosink.hostname = localhost agent6.sinks.avrosink.port = 4000 agent6.channels.memorychannel.type = memory agent6.channels.memorychannel.capacity = 1000 agent6.channels.memorychannel.transactionCapacity = 100 agent6.sources.avrosource.channels = memorychannel agent6.sinks.avrosink.channel = memorychannel
agent3.conf
file created earlier, that is, with an Avro source and a file sink:$ flume-ng client –conf conf –conf-file agent3.conf agent3
$ flume-ng client –conf conf –conf-file agent6.conf agent6
$ flume-ng avro-client –H localhost –p 4000 –F /home/hadoop/message $ flume-ng avro-client -H localhost -p 2000 -F /home/hadoop/message2
Firstly, we defined a new agent with an Avro source and also an Avro sink. We've not used this sink before; instead of writing the events to a local location or HDFS, this sink sends the events to a remote source using Avro.
We start an instance of this new agent and then also start an instance of agent3
we used earlier. Recall this agent has an Avro source and a file roll sink. We configure the Avro sink in the first agent to point to the host and port of the Avro sink in the second and by doing so, build a data-routing chain.
With both agents running, we then use the Avro client to send a file to each and confirm that both appear in the file location configured as the destination for the agent3
sink.
This isn't just technical capability for its own sake. This capability is the building block that allows Flume to build arbitrarily complex distributed event collection networks. Instead of one copy of each agent, think of multiple agents of each type feeding events into the next link in the chain, which acts as an event aggregation point.
3.145.40.189