Fluentd
Search…
Data Collection with Hadoop (HDFS)
This article explains how to use Fluentd's WebHDFS Output plugin to aggregate semi-structured logs into Hadoop HDFS.
Background
​Fluentd is an advanced open-source log collector originally developed at Treasure Data, Inc. Fluentd is specifically designed to solve the big-data log collection problem. A lot of users are using Fluentd with MongoDB, and have found that it doesn't scale well for now.
HDFS (Hadoop) is a natural alternative for storing and processing a huge amount of data. It supports an HTTP interface called WebHDFS in addition to its Java library.
Architecture
The figure below shows the high-level architecture:
HTTP-to-HDFS Overview
Install
For simplicity, this article will describe how to set up a one-node configuration. Please install the following software on the same node:
- Apache HDFS
The WebHDFS Output plugin is included in the latest version of Fluentd's deb/rpm package (v1.1.10 or later). If you want to use RubyGems to install the plugin, please use
gem install fluent-plugin-webhdfs.Fluentd Configuration
Let's start configuring Fluentd. If you used the deb/rpm package, Fluentd's config file is located at
/etc/td-agent/td-agent.conf. Otherwise, it is located at /etc/fluentd/fluentd.conf.HTTP Input
For the input source, we will set up Fluentd to accept records from HTTP. The Fluentd configuration file should look like this:
1
<source>
2
@type http
3
port 8888
4
</source>
Copied!
WebHDFS Output
The output destination will be WebHDFS. The output configuration should look like this:
1
<match hdfs.*.*>
2
@type webhdfs
3
host namenode.your.cluster.local
4
port 50070
5
path "/log/%Y%m%d_%H/access.log.#{Socket.gethostname}"
6
<buffer>
7
flush_interval 10s
8
</buffer>
9
</match>
Copied!
The
<match> section specifies the regexp used to look for matching tags. If a tag in a log is matched, the respective match configuration is used (i.e. the log is routed accordingly).The
flush_interval parameter specifies how often the data is written to HDFS. An append operation is used to append the incoming data to the file specified by the path parameter.Placeholders for both time and hostname can be used with the
path parameter. This prevents multiple Fluentd instances from appending data to the same file, which must be avoided for append operations.Other options specify HDFS's NameNode host and port.
HDFS Configuration
Append operations are not enabled by default. Please put these configurations into your
hdfs-site.xml file and restart the whole cluster:1
<property>
2
<name>dfs.webhdfs.enabled</name>
3
<value>true</value>
4
</property>
5
​
6
<property>
7
<name>dfs.support.append</name>
8
<value>true</value>
9
</property>
10
​
11
<property>
12
<name>dfs.support.broken.append</name>
13
<value>true</value>
14
</property>
Copied!
Please confirm that the HDFS user has the write access to the
path specified as the WebHDFS output.Test
To test the configuration, just post the JSON to Fluentd (we use the
curl command in this example). Sending a USR1 signal flushes Fluentd's buffer into WebHDFS:1
$ curl -X POST -d 'json={"action":"login","user":2}' \
2
http://localhost:8888/hdfs.access.test
3
$ kill -USR1 `cat /var/run/td-agent/td-agent.pid`
Copied!
We can then access HDFS to see the stored data:
1
$ sudo -u hdfs hadoop fs -lsr /log/
2
drwxr-xr-x - 1 supergroup 0 2012-10-22 09:40 /log/20121022_14/access.log.dev
Copied!
Conclusion
Fluentd with WebHDFS makes the realtime log collection simple, robust and scalable! @tagomoris has already been using this plugin to collect 20,000 msgs/sec, 1.5 TB/day without any major problems for several months now.
Learn More
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 1yr ago