Fluentd
Search…
Apache To Riak
This article explains how to use Fluentd's Riak Output plugin (out_riak) to aggregate semi-structured logs in real-time.
Could not load image
Prerequisites
- 1.An OSX or Linux machine
- 2.
- 3.Riak is installed
- 4.An Apache web server log
Installing the Fluentd Riak Output Plugin
Rubygems Users
Rubygems users can run the command below to install the plugin:
1
$ gem install fluent-plugin-riak
Copied!
td-agent Users
If you are using td-agent, run following command to install the Riak output plugin.
- td-agent v2:
/usr/sbin/td-agent-gem install fluent-plugin-riak - td-agent v1: `/usr/lib/fluent/ruby/bin/fluent-gem installfluent-plugin-riak
Configuring Fluentd
Create a configuration file called
fluent.conf and add the following lines:1
<source>
2
@type tail
3
format apache2
4
path /var/log/apache2/access_log
5
pos_file /var/log/fluentd/apache2.access_log.pos
6
tag riak.apache
7
</source>
8
​
9
<match riak.**>
10
@type riak
11
buffer_type memory
12
flush_interval 5s
13
retry_limit 5
14
retry_wait 1s
15
nodes localhost:8087 # Assumes Riak is running locally on port 8087
16
</match>
Copied!
The
<source>...</source> section tells Fluentd to tail an Apache2-formatted log file located at /var/log/apache2/access_log. Each line is parsed as an Apache access log event and tagged with the riak.apache label.The
<match riak.**>...</match> section tells Fluentd to look for events whose tags start with riak. and send all matches to a Riak node located at localhost:8087. You can send events to multiple nodes by writing nodes host1 host2 host3 instead.Testing
Launch Fluentd with the following command:
1
$ fluentd -c fluentd.conf
Copied!
Please confirm that you have the file access permissions to (1) read the Apache log file and (2) write to `/var/log/fluentd/apache2.access_log.pos` (sudo-ing might help).
You should now see data coming into your Riak cluster. We can make sure that everything is running smoothly by hitting Riak's HTTP API:
1
$ curl http://localhost:8098/buckets/fluentlog/keys?keys=true
2
{"keys":["2014-01-23-d30b0698-b9de-4290-b8be-a66555497078", ...]}
3
$ curl http://localhost:8098/buckets/fluentlog/keys/2014-01-23-d30b0698-b9de-4290-b8be-a66555497078
4
[
5
{
6
"tag": "riak.apache",
7
"time": "2004-03-08T01:23:54Z",
8
"host": "64.242.88.10",
9
"user": null,
10
"method": "GET",
11
"path": "/twiki/bin/statistics/Main",
12
"code": 200,
13
"size": 808,
14
"referer": null,
15
"agent": null
16
}
17
]
Copied!
There it is! (the response JSON is formatted for readability)
Learn More
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 2yr ago