Apache To Riak

This article explains how to use Fluentd's Riak Output plugin (out_riak) to aggregate semi-structured logs in real-time.
Prerequisites
1.
An OSX or Linux machine
2.
Fluentd is installed (installation guide)
3.
Riak is installed
4.
An Apache web server log
Installing the Fluentd Riak Output Plugin
The Riak output plugin is used to output data from a Fluentd node to a Riak node.
Rubygems Users
Rubygems users can run the command below to install the plugin:
1
$ gem install fluent-plugin-riak
Copied!
td-agent Users
If you are using td-agent, run following command to install the Riak output plugin.
td-agent v2: /usr/sbin/td-agent-gem install fluent-plugin-riak
td-agent v1: `/usr/lib/fluent/ruby/bin/fluent-gem install
fluent-plugin-riak
Configuring Fluentd
Create a configuration file called fluent.conf and add the following lines:
1
<source>
2
  @type tail
3
  format apache2
4
  path /var/log/apache2/access_log
5
  pos_file /var/log/fluentd/apache2.access_log.pos
6
  tag riak.apache
7
</source>
8
​
9
<match riak.**>
10
  @type riak
11
  buffer_type memory
12
  flush_interval 5s
13
  retry_limit 5
14
  retry_wait 1s
15
  nodes localhost:8087 # Assumes Riak is running locally on port 8087
16
</match>
Copied!
The <source>...</source> section tells Fluentd to tail an Apache2-formatted log file located at /var/log/apache2/access_log. Each line is parsed as an Apache access log event and tagged with the riak.apache label.
The <match riak.**>...</match> section tells Fluentd to look for events whose tags start with riak. and send all matches to a Riak node located at localhost:8087. You can send events to multiple nodes by writing nodes host1 host2 host3 instead.
Testing
Launch Fluentd with the following command:
1
$ fluentd -c fluentd.conf
Copied!
Please confirm that you have the file access permissions to (1) read the Apache log file and (2) write to `/var/log/fluentd/apache2.access_log.pos` (sudo-ing might help).
You should now see data coming into your Riak cluster. We can make sure that everything is running smoothly by hitting Riak's HTTP API:
1
$ curl http://localhost:8098/buckets/fluentlog/keys?keys=true
2
{"keys":["2014-01-23-d30b0698-b9de-4290-b8be-a66555497078", ...]}
3
$ curl http://localhost:8098/buckets/fluentlog/keys/2014-01-23-d30b0698-b9de-4290-b8be-a66555497078
4
[
5
  {
6
    "tag": "riak.apache",
7
    "time": "2004-03-08T01:23:54Z",
8
    "host": "64.242.88.10",
9
    "user": null,
10
    "method": "GET",
11
    "path": "/twiki/bin/statistics/Main",
12
    "code": 200,
13
    "size": 808,
14
    "referer": null,
15
    "agent": null
16
  }
17
]
Copied!
There it is! (the response JSON is formatted for readability)
Learn More
​Fluentd Architecture​
​Fluentd Get Started​
​Riak Output Plugin​
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.

Store Apache Logs into MongoDB

Store Apache Logs into Amazon S3

Last modified 2yr ago

Copy link

Contents

Prerequisites

Installing the Fluentd Riak Output Plugin