Fluentd
Search…
Stream Processing with Norikra
Background
​Fluentd is an advanced open-source log collector originally developed at Treasure Data, Inc. Fluentd is not only a log collector, but also an all-purpose stream processing platform. Plugins can be written to handle many kinds of events.
However, Fluentd is not primarily designed for stream processing. We must restart Fluentd after making modifications to its configuration/code, making it unsuitable for running both short-span (seconds or minutes) calculations and long-span (hours or days) calculations. If we restart Fluentd to perform a short-span calculation, all existing internal statuses of short and long span calculations are lost. For large scale stream processing platforms, code/processes must be added/removed without any such losses.
​Norikra is an open-source stream processing server based on Esper by EsperTech. It allows you to subscribe/unsubscribe to data streams anytime and add/remove SQL queries anytime. Norikra is written by @tagomoris, a committer of the Fluentd project.
This article will show you how to integrate Fluentd, Norikra, and the Fluentd norikra plugin to create a robust stream data processing platform.
Architecture
The figure below shows the high-level architecture:
Fluentd + Norikra Overview
Installation
For the sake of simplicity, this guide will set up a one-node configuration. Please install the following on the same node:
Installing Fluentd and fluentd-plugin-norikra
Fluentd can be installed through RubyGems or via deb/rpm packages.
fluent-plugin-norikra can be installed with the gem install fluent-plugin-norikra (or fluent-gem) command.Installing Norikra
Norikra requires JRuby. You can download the JRuby binary directly from the official site and export the PATH of
JRUBY_INSTALL_DIRECTORY/bin.Once JRuby is set up, install Norikra:
1
jgem install norikra
Copied!
Verify Installation
We'll start the Norikra server after installation. The
norikra start command will launch the Norikra server in your console.1
....
2
2014-05-20 20:36:01 +0900 [INFO] : Loading UDF plugins
3
2014-05-20 20:36:01 +0900 [INFO] : RPC server 0.0.0.0:26571, 2 threads
4
2014-05-20 20:36:01 +0900 [INFO] : WebUI server 0.0.0.0:26578, 2 threads
5
2014-05-20 20:36:01 +0900 [INFO] : Norikra server started.
Copied!
Fluentd Configuration
We'll now configure Fluentd. If you used the deb/rpm package, Fluentd's config file is located at
/etc/td-agent/td-agent.conf. Otherwise, it is located at /etc/fluentd/fluentd.conf.HTTP Input
For the input source, we will set up Fluentd to accept records from HTTP. The Fluentd configuration file should look like this:
1
<source>
2
@type http
3
port 8888
4
</source>
Copied!
Norikra Output
The output destination will be Norikra. The output configuration should look like this:
1
<match data.*>
2
@type norikra
3
norikra localhost:26571
4
target_map_tag true
5
remove_tag_prefix data
6
​
7
<default>
8
include *
9
exclude time
10
</default>
11
</match>
Copied!
The
<match> section specifies the glob pattern used to look for the matching tags. If the tag of a log is matched, the respective match configuration is used (i.e. the log is routed accordingly).The
norikra attribute specifies the Norikra server's RPC host and port (default: 26571). By target_map_tag true and remove_tag_prefix data, out_norikra handle the rest of tags (e.g. foo for data.foo) as the target, which is the name of the set of events as same as table name of RDBMS.The
<default> section specifies which fields are sent to the Norikra server. We can also specify these sets per target with <target NAME>...</target>. For more details, refer to fluent-plugin-norikra.Test
To test the configuration, just post the JSON to Fluentd (we use the
curl command in this example):1
$ curl -X POST -d 'json={"action":"login","user":2}' \
2
http://localhost:8888/data.access
Copied!
Norikra's console log will show that Fluentd has opened the target
access and sent a message with fields of action and user.1
2014-05-20 20:43:22 +0900 [INFO] : opening target, target:"access", fields:{}, auto_field:true
2
2014-05-20 20:43:23 +0900 [INFO] : opening lazy target, target:#<Norikra::Target:0x69c04611 @last_modified=nil, @fields={}, @name="access", @auto_field=true>
3
2014-05-20 20:43:23 +0900 [INFO] : target successfully opened (snip)
Copied!
We can check its fields with
norikra-client command from the console that has the PATH to JRuby:1
$ norikra-client target list
2
TARGET AUTO_FIELD
3
access true
4
1 targets found.
5
$ norikra-client field list access
6
FIELD TYPE OPTIONAL
7
action string false
8
user integer false
9
2 fields found.
Copied!
Registering Queries and Fetching Outputs
We can add queries on opened targets via the WebUI or CLI. The following query (just SQL!) counts the number of events with a non-zero
user per 10 second interval, with a 'group by' action:1
SELECT
2
action,
3
count(*) AS c
4
FROM access.win:time_batch(10 sec)
5
WHERE user != 0
6
GROUP BY action
Copied!
To register a query, issue
norikra-client query add on the CLI:1
$ norikra-client query add test_query "SELECT action, count(*) AS c FROM access.win:time_batch(10 sec) WHERE user != 0 GROUP BY action"
2
$ norikra-client query list
3
NAME GROUP TARGETS QUERY
4
test_query default access SELECT action, count(*) AS c FROM access.win:time_batch(10 sec) WHERE user != 0 GROUP BY action
5
1 queries found.
Copied!
Once the query has been registered, post the events that you want:
1
$ curl -X POST -d 'json={"action":"login","user":2}' \
2
http://localhost:8888/data.access
3
$ curl -X POST -d 'json={"action":"login","user":0}' \
4
http://localhost:8888/data.access
5
$ curl -X POST -d 'json={"action":"write","user":2}' \
6
http://localhost:8888/data.access
7
$ curl -X POST -d 'json={"action":"save","user":2}' \
8
http://localhost:8888/data.access
9
$ curl -X POST -d 'json={"action":"logout","user":2}' \
10
http://localhost:8888/data.access
11
$ curl -X POST -d 'json={"action":"logout","user":0}' \
12
http://localhost:8888/data.access
13
$ curl -X POST -d 'json={"action":"login","user":2}' \
14
http://localhost:8888/data.access
Copied!
And fetch output events from this
test_query query:1
$ norikra-client event fetch test_query
2
{"time":"2014/05/20 21:00:24","c":1,"action":"logout"}
3
{"time":"2014/05/20 21:00:24","c":1,"action":"save"}
4
{"time":"2014/05/20 21:00:24","c":1,"action":"write"}
5
{"time":"2014/05/20 21:00:24","c":2,"action":"login"}
6
{"time":"2014/05/20 21:00:34","c":0,"action":"logout"}
7
{"time":"2014/05/20 21:00:34","c":0,"action":"save"}
8
{"time":"2014/05/20 21:00:34","c":0,"action":"write"}
9
{"time":"2014/05/20 21:00:34","c":0,"action":"login"}
10
$
Copied!
If posts are done in 10 seconds, this query calculates all the events in first 10 seconds, and counts events per
action for events with user != 0 only, and outputs events at "2014/05/20 21:00:24". At "2014/05/20 21:00:34", just after next 10 seconds, this query reports that no events arrived (These are teardown records, and reported only once).Conclusion
We can create a stream data processing platform without any schema definitions, using Fluentd and Norikra. This platform enables an agile stream processing environment that can handle real workloads.
Learn More
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 11mo ago