Simple Stream Processing with Fluentd

Search…
Sometimes, you need to transform the data stream in a certain way. For example, you might want to extract a portion of data for error reporting, or need to append additional information to events for later inspection.
This article explains the common data manipulation techniques in detail.
How to Filter Events by Fields
​filter_grep is a built-in plugin that allows to filter the data stream using regular expressions.
Suppose you are managing a web service, and try to monitor the access logs using Fluentd. In this case, an event in the data stream will look like:
1
{
2
  "host": "192.168.1.1",
3
  "method": "GET",
4
  "path": "/index.html",
5
  "code": 200,
6
  "size": 2344,
7
  "referer": null
8
}
Copied!
Let's filter out all the successful responses with 2xx status codes so that we can easily inspect if any error has occurred in the service:
1
<filter apache.**>
2
  @type grep
3
  <exclude>
4
    key code
5
    pattern ^2\d\d$
6
  </exclude>
7
</filter>
Copied!
You can also filter the data using multiple fields. The following configuration will keep all 5xx server errors, except those coming from the test directory:
1
<filter apache.**>
2
  @type grep
3
  <regexp>
4
    key code
5
    pattern ^5\d\d$
6
  </regexp>
7
  <exclude>
8
    key path
9
    pattern ^/test/
10
  </exclude>
11
</filter>
Copied!
How to Inject Custom Fields into Events
​filter_record_transformer is a built-in plugin that enables it to inject arbitrary data into events.
Suppose you are running a web service on multiple web servers, and you want to record which server handled each request. This can be implemented trivially using this plugin:
1
<filter apache.**>
2
  @type record_transformer
3
  <record>
4
    server "${hostname}"
5
  </record>
6
</filter>
Copied!
This will produce an event like this:
1
{
2
  "host": "192.168.1.1",
3
  "method": "GET",
4
  "path": "/index.html",
5
  "code": 200,
6
  "size": 2344,
7
  "referer": null,
8
  "server": "app1"
9
}
Copied!
Note that ${hostname} is a predefined variable supplied by the plugin. You can also define a custom variable, or even evaluate arbitrary ruby expressions. For more details, see record_transformer.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Data Collection with Hadoop (HDFS)
Stream Processing with Norikra
Last modified 4mo ago
Copy link
Contents
How to Filter Events by Fields
How to Inject Custom Fields into Events