gem install fluent-plugin-webhdfs
./etc/td-agent/td-agent.conf
. Otherwise, it is located at /etc/fluentd/fluentd.conf
.<match>
section specifies the regexp used to look for matching tags. If a tag in a log is matched, the respective match
configuration is used (i.e. the log is routed accordingly).flush_interval
parameter specifies how often the data is written to HDFS. An append operation is used to append the incoming data to the file specified by the path
parameter.path
parameter. This prevents multiple Fluentd instances from appending data to the same file, which must be avoided for append operations.hdfs-site.xml
file and restart the whole cluster:path
specified as the WebHDFS output.curl
command in this example). Sending a USR1
signal flushes Fluentd's buffer into WebHDFS: