Stream Processing with Norikra
Last updated
Was this helpful?
Last updated
Was this helpful?
This article explains how to use and to create a SQL-based realtime complex event processing platform.
is an advanced open-source log collector originally developed at . Fluentd is not only a log collector, but also an all-purpose stream processing platform. Plugins can be written to handle many kinds of events.
However, Fluentd is not primarily designed for stream processing. We must restart Fluentd after making modifications to its configuration/code, making it unsuitable for running both short-span (seconds or minutes) calculations and long-span (hours or days) calculations. If we restart Fluentd to perform a short-span calculation, all existing internal statuses of short and long span calculations are lost. For large scale stream processing platforms, code/processes must be added/removed without any such losses.
is an open-source stream processing server based on by . It allows you to subscribe/unsubscribe to data streams anytime and add/remove SQL queries anytime. is written by , a committer of the Fluentd project.
This article will show you how to integrate , , and the to create a robust stream data processing platform.
The figure below shows the high-level architecture:
The following software/services are required to be set up correctly:
You can install Fluentd via major packaging systems.
If out_norikra
(fluent-plugin-norikra) is not installed yet, please install it manually.
Once JRuby is set up, install Norikra:
We'll start the Norikra server after installation. The norikra start
command will launch the Norikra server in your console.
We'll now configure Fluentd. If you used the deb/rpm package, Fluentd's config file is located at /etc/fluent/fluentd.conf
.
For the input source, we will set up Fluentd to accept records from HTTP. The Fluentd configuration file should look like this:
The output destination will be Norikra. The output configuration should look like this:
The <match>
section specifies the glob pattern used to look for the matching tags. If the tag of a log is matched, the respective match
configuration is used (i.e. the log is routed accordingly).
The norikra
attribute specifies the Norikra server's RPC host and port (default: 26571
). By target_map_tag true
and remove_tag_prefix data
, out_norikra
handle the rest of tags (e.g. foo
for data.foo
) as the target, which is the name of the set of events as same as table name of RDBMS.
To test the configuration, just post the JSON to Fluentd (we use the curl
command in this example):
Norikra's console log will show that Fluentd has opened the target access
and sent a message with fields of action
and user
.
We can check its fields with norikra-client
command from the console that has the PATH
to JRuby:
We can add queries on opened targets via the WebUI or CLI. The following query (just SQL!) counts the number of events with a non-zero user
per 10 second interval, with a 'group by' action
:
To register a query, issue norikra-client query add
on the CLI:
Once the query has been registered, post the events that you want:
And fetch output events from this test_query
query:
If posts are done in 10 seconds, this query calculates all the events in first 10 seconds, and counts events per action
for events with user != 0
only, and outputs events at "2014/05/20 21:00:24". At "2014/05/20 21:00:34", just after next 10 seconds, this query reports that no events arrived (These are teardown records, and reported only once).
We can create a stream data processing platform without any schema definitions, using Fluentd and Norikra. This platform enables an agile stream processing environment that can handle real workloads.
See section how to install fluent-plugin-norikra on your environment.
Norikra requires JRuby. You can download the JRuby binary directly from the and export the PATH of JRUBY_INSTALL_DIRECTORY/bin
.
You can also check the current Norikra's status via the WebUI ().
The <default>
section specifies which fields are sent to the Norikra server. We can also specify these sets per target with <target NAME>...</target>
. For more details, refer to .
If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.