Fluentd
0.12
0.12
  • Introduction
  • Overview
    • Getting Started
    • Installation
    • Life of a Fluentd event
    • Support
    • FAQ
  • Use Cases
    • Centralized App Logging
    • Monitoring Service Logs
    • Data Analytics
    • Connecting to Data Storages
    • Stream Processing
    • Windows Event Collection
    • IoT Data Logger
  • Configuration
    • Config File Syntax
    • Routing Examples
    • Recipes
  • Deployment
    • Logging
    • Monitoring
    • Signals
    • RPC
    • High Availability Config
    • Failure Scenarios
    • Performance Tuning
    • Plugin Management
    • Trouble Shooting
    • Secure Forwarding
    • Fluentd UI
    • Command Line Option
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Input Plugins
    • tail
    • forward
    • secure_forward
    • udp
    • tcp
    • http
    • unix
    • syslog
    • exec
    • scribe
    • multiprocess
    • dummy
    • Others
  • Output Plugins
    • file
    • s3
    • kafka
    • forward
    • secure_forward
    • exec
    • exec_filter
    • copy
    • geoip
    • roundrobin
    • stdout
    • null
    • webhdfs
    • splunk
    • mongo
    • mongo_replset
    • relabel
    • rewrite_tag_filter
    • Others
  • Buffer Plugins
    • memory
    • file
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
  • Developer
    • Plugin Development
    • Community
    • Mailing List
    • Source Code
    • Bug Tracking
    • ChangeLog
    • Logo
  • Articles
    • Store Apache Logs into MongoDB
    • Apache To Riak
    • Store Apache Logs into Amazon S3
    • Before Install
    • Cep Norikra
    • Collect Glusterfs Logs
    • Common Log Formats
    • Docker Logging Efk Compose
    • Docker Logging
    • Filter Modify Apache
    • Forwarding Over Ssl
    • Free Alternative To Splunk By Fluentd
    • Data Collection to Hadoop (HDFS)
    • Data Analytics with Treasure Data
    • Install By Chef
    • Install By Deb
    • Install By Dmg
    • Install By Gem
    • Install By Rpm
    • Install From Source
    • Install On Beanstalk
    • Install On Heroku
    • Java
    • Kinesis Stream
    • Kubernetes Fluentd
    • Monitoring by Prometheus
    • Monitoring by Rest Api
    • Nodejs
    • Performance Tuning Multi Process
    • Performance Tuning Single Process
    • Perl
    • Php
    • Python
    • Quickstart
    • Raspberrypi Cloud Data Logger
    • Recipe Apache Logs To Elasticsearch
    • Recipe Apache Logs To Mongo
    • Recipe Apache Logs To S3
    • Recipe Apache Logs To Treasure Data
    • Recipe Cloudstack To Mongodb
    • Recipe Csv To Elasticsearch
    • Recipe Csv To Mongo
    • Recipe Csv To S3
    • Recipe Csv To Treasure Data
    • Recipe Http Rest Api To Elasticsearch
    • Recipe Http Rest Api To Mongo
    • Recipe Http Rest Api To S3
    • Recipe Http Rest Api To Treasure Data
    • Recipe Json To Elasticsearch
    • Recipe Json To Mongo
    • Recipe Json To S3
    • Recipe Json To Treasure Data
    • Recipe Nginx To Elasticsearch
    • Recipe Nginx To Mongo
    • Recipe Nginx To S3
    • Recipe Nginx To Treasure Data
    • Recipe Syslog To Elasticsearch
    • Recipe Syslog To Mongo
    • Recipe Syslog To S3
    • Recipe Syslog To Treasure Data
    • Recipe Tsv To Elasticsearch
    • Recipe Tsv To Mongo
    • Recipe Tsv To S3
    • Recipe Tsv To Treasure Data
    • Ruby
    • Scala
    • Splunk Like Grep And Alert Email
Powered by GitBook
On this page
  • Background
  • Architecture
  • Installation
  • Installing Fluentd and fluentd-plugin-norikra
  • Installing Norikra
  • Verify Installation
  • Fluentd Configuration
  • HTTP Input
  • Norikra Output
  • Test
  • Registering Queries and Fetching Outputs
  • Conclusion
  • Learn More

Was this helpful?

  1. Articles

Cep Norikra

PreviousBefore InstallNextCollect Glusterfs Logs

Last updated 5 years ago

Was this helpful?

This article explains how to use and to create a SQL-based real-time complex event processing platform.

Background

is an advanced open-source log collector originally developed at . Fluentd is not only a log collector, but also an all-purpose stream processing platform. Plugins can be written to handle many kinds of events.

However, Fluentd is not primarily designed for stream processing. We must restart Fluentd after making modifications to its configuration/code, making it unsuitable for running both short-span (seconds or minutes) calculations and long-span (hours or days) calculations. If we restart Fluentd to perform a short-span calculation, all existing internal statuses of short and long span calculations are lost. For large scale stream processing platforms, code/processes must be added/removed without any such losses.

is an open-source stream processing server based on by . It allows you to subscribe/unsubscribe to data streams anytime and add/remove SQL queries anytime. is written by , a committer of the Fluentd project.

This article will show you how to integrate , , and the to create a robust stream data processing platform.

Architecture

The figure below shows the high-level architecture.

Installation

For the sake of simplicity, this guide will set up a one-node configuration. Please install the following on the same node:

Installing Fluentd and fluentd-plugin-norikra

Fluentd can be installed through rubygems or via deb/rpm packages.

fluent-plugin-norikra can be installed with the gem install fluent-plugin-norikra (or fluent-gem) command.

Installing Norikra

Once JRuby has been installed, simply entering jgem install norikra will install Norikra.

Verify Installation

We'll start the Norikra server after installation. The norikra start command will launch the Norikra server in your console.

....
2014-05-20 20:36:01 +0900 [INFO] : Loading UDF plugins
2014-05-20 20:36:01 +0900 [INFO] : RPC server 0.0.0.0:26571, 2 threads
2014-05-20 20:36:01 +0900 [INFO] : WebUI server 0.0.0.0:26578, 2 threads
2014-05-20 20:36:01 +0900 [INFO] : Norikra server started.

Fluentd Configuration

We'll now configure Fluentd. If you used the deb/rpm package, Fluentd's config file is located at /etc/td-agent/td-agent.conf. Otherwise, it is located at /etc/fluentd/fluentd.conf.

HTTP Input

For the input source, we will set up Fluentd to accept records from HTTP. The Fluentd configuration file should look like this:

<source>
  @type http
  port 8888
</source>

Norikra Output

The output destination will be Norikra. The output configuration should look like this:

<match data.*>
  @type    norikra
  norikra localhost:26571
  target_map_tag    true
  remove_tag_prefix data

  <default>
    include *
    exclude time
  </default>
</match>

The match section specifies the glob pattern used to look for matching tags. If a matching tag is found in a log, then the config inside <match>...</match> is used (i.e. the log is routed according to the config inside).

The norikra attribute specifies the Norikra server's RPC host and port ('26571' is the default port of Norikra RPC protocol). By target_map_tag true and remove_tag_prefix data, out_norikra handle the rest of tags (ex: 'foo' for 'data.foo') as target, which is a name of set of events as same as table name of RDBMS.

Test

To test the configuration, just post the JSON to Fluentd (we use the curl command in this example).

$ curl -X POST -d 'json={"action":"login","user":2}' \
  http://localhost:8888/data.access

Norikra's console log will show that Fluentd has opened the target access and sent a message with fields of action and user.

2014-05-20 20:43:22 +0900 [INFO] : opening target, target:"access", fields:{}, auto_field:true
2014-05-20 20:43:23 +0900 [INFO] : opening lazy target, target:#<Norikra::Target:0x69c04611 @last_modified=nil, @fields={}, @name="access", @auto_field=true>
2014-05-20 20:43:23 +0900 [INFO] : target successfully opened (snip)

We can check its fields with the norikra-client command (from console that has the PATH to JRuby).

$ norikra-client target list
TARGET  AUTO_FIELD
access  true
1 targets found.
$ norikra-client field list access
FIELD   TYPE    OPTIONAL
action  string  false
user    integer false
2 fields found.

Registering Queries and Fetching Outputs

We can add queries on opened targets via the WebUI or CLI. The following query (just SQL!) counts the number of events with a non-zero user per 10 second interval, with a 'group by' action.

SELECT
  action,
  count(*) AS c
FROM access.win:time_batch(10 sec)
WHERE user != 0
GROUP BY action

To register a query, issue norikra-client query add on the CLI.

$ norikra-client query add test_query "SELECT action, count(*) AS c FROM access.win:time_batch(10 sec) WHERE user != 0 GROUP BY action"
$ norikra-client query list
NAME    GROUP   TARGETS QUERY
test_query  default access  SELECT action, count(*) AS c FROM access.win:time_batch(10 sec) WHERE user != 0 GROUP BY action
1 queries found.

Once the query has been registered, post the events that you want.

$ curl -X POST -d 'json={"action":"login","user":2}' \
  http://localhost:8888/data.access
$ curl -X POST -d 'json={"action":"login","user":0}' \
  http://localhost:8888/data.access
$ curl -X POST -d 'json={"action":"write","user":2}' \
  http://localhost:8888/data.access
$ curl -X POST -d 'json={"action":"save","user":2}' \
  http://localhost:8888/data.access
$ curl -X POST -d 'json={"action":"logout","user":2}' \
  http://localhost:8888/data.access
$ curl -X POST -d 'json={"action":"logout","user":0}' \
  http://localhost:8888/data.access
$ curl -X POST -d 'json={"action":"login","user":2}' \
  http://localhost:8888/data.access

And fetch output events from this test_query query.

$ norikra-client event fetch test_query
{"time":"2014/05/20 21:00:24","c":1,"action":"logout"}
{"time":"2014/05/20 21:00:24","c":1,"action":"save"}
{"time":"2014/05/20 21:00:24","c":1,"action":"write"}
{"time":"2014/05/20 21:00:24","c":2,"action":"login"}
{"time":"2014/05/20 21:00:34","c":0,"action":"logout"}
{"time":"2014/05/20 21:00:34","c":0,"action":"save"}
{"time":"2014/05/20 21:00:34","c":0,"action":"write"}
{"time":"2014/05/20 21:00:34","c":0,"action":"login"}
$

If posts are done in 10 seconds, this query calculates all events in first 10 seconds, and counts events per action for events with user != 0 only, and outputs events at "2014/05/20 21:00:24". At "2014/05/20 21:00:34", just after next 10 seconds, this query reports that no events arrived (This is teardown records, and reported only once).

Conclusion

We can create a stream data processing platform without any schema definitions, using Fluentd and Norikra. This platform enables an agile stream processing environment that can handle real workloads.

Learn More

  • Fluentd Get Started

Norikra requires JRuby. You can download the JRuby binary directly from the and export the PATH of JRUBY_INSTALL_DIRECTORY/bin.

You can also check the current Norikra's status via the WebUI ().

The <default>...</default> section specifies which fields are sent to the Norikra server. We can also specify these sets per target with <target NAME>...</target>. For information on the additional options available, please refer to the .

If this article is incorrect or outdated, or omits critical information, please . is a open source project under . All components are available under the Apache 2 License.

Fluentd
Fluentd Norikra Plugin
Norikra
Debian Package
RPM Package
Ruby gem
official site
http://localhost:26578/
fluent-plugin-norikra documentation
Fluentd Architecture
Norikra: Query Syntax
Norikra: Query Examples
Slides: fluent-plugin-norikra
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
Fluentd
Norikra
Fluentd
Treasure Data, Inc
Norikra
Esper
EsperTech
Norikra
tagomoris
Fluentd
Norikra
Fluentd norikra plugin