Fluentd
0.12
0.12
  • Introduction
  • Overview
    • Getting Started
    • Installation
    • Life of a Fluentd event
    • Support
    • FAQ
  • Use Cases
    • Centralized App Logging
    • Monitoring Service Logs
    • Data Analytics
    • Connecting to Data Storages
    • Stream Processing
    • Windows Event Collection
    • IoT Data Logger
  • Configuration
    • Config File Syntax
    • Routing Examples
    • Recipes
  • Deployment
    • Logging
    • Monitoring
    • Signals
    • RPC
    • High Availability Config
    • Failure Scenarios
    • Performance Tuning
    • Plugin Management
    • Trouble Shooting
    • Secure Forwarding
    • Fluentd UI
    • Command Line Option
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Input Plugins
    • tail
    • forward
    • secure_forward
    • udp
    • tcp
    • http
    • unix
    • syslog
    • exec
    • scribe
    • multiprocess
    • dummy
    • Others
  • Output Plugins
    • file
    • s3
    • kafka
    • forward
    • secure_forward
    • exec
    • exec_filter
    • copy
    • geoip
    • roundrobin
    • stdout
    • null
    • webhdfs
    • splunk
    • mongo
    • mongo_replset
    • relabel
    • rewrite_tag_filter
    • Others
  • Buffer Plugins
    • memory
    • file
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
  • Developer
    • Plugin Development
    • Community
    • Mailing List
    • Source Code
    • Bug Tracking
    • ChangeLog
    • Logo
  • Articles
    • Store Apache Logs into MongoDB
    • Apache To Riak
    • Store Apache Logs into Amazon S3
    • Before Install
    • Cep Norikra
    • Collect Glusterfs Logs
    • Common Log Formats
    • Docker Logging Efk Compose
    • Docker Logging
    • Filter Modify Apache
    • Forwarding Over Ssl
    • Free Alternative To Splunk By Fluentd
    • Data Collection to Hadoop (HDFS)
    • Data Analytics with Treasure Data
    • Install By Chef
    • Install By Deb
    • Install By Dmg
    • Install By Gem
    • Install By Rpm
    • Install From Source
    • Install On Beanstalk
    • Install On Heroku
    • Java
    • Kinesis Stream
    • Kubernetes Fluentd
    • Monitoring by Prometheus
    • Monitoring by Rest Api
    • Nodejs
    • Performance Tuning Multi Process
    • Performance Tuning Single Process
    • Perl
    • Php
    • Python
    • Quickstart
    • Raspberrypi Cloud Data Logger
    • Recipe Apache Logs To Elasticsearch
    • Recipe Apache Logs To Mongo
    • Recipe Apache Logs To S3
    • Recipe Apache Logs To Treasure Data
    • Recipe Cloudstack To Mongodb
    • Recipe Csv To Elasticsearch
    • Recipe Csv To Mongo
    • Recipe Csv To S3
    • Recipe Csv To Treasure Data
    • Recipe Http Rest Api To Elasticsearch
    • Recipe Http Rest Api To Mongo
    • Recipe Http Rest Api To S3
    • Recipe Http Rest Api To Treasure Data
    • Recipe Json To Elasticsearch
    • Recipe Json To Mongo
    • Recipe Json To S3
    • Recipe Json To Treasure Data
    • Recipe Nginx To Elasticsearch
    • Recipe Nginx To Mongo
    • Recipe Nginx To S3
    • Recipe Nginx To Treasure Data
    • Recipe Syslog To Elasticsearch
    • Recipe Syslog To Mongo
    • Recipe Syslog To S3
    • Recipe Syslog To Treasure Data
    • Recipe Tsv To Elasticsearch
    • Recipe Tsv To Mongo
    • Recipe Tsv To S3
    • Recipe Tsv To Treasure Data
    • Ruby
    • Scala
    • Splunk Like Grep And Alert Email
Powered by GitBook
On this page
  • Example Configurations
  • Parameters
  • <record> directive
  • enable_ruby (optional)
  • auto_typecast (optional)
  • renew_record (optional)
  • renew_time_key (optional, string type)
  • keep_keys (optional, array type)
  • remove_keys (optional, array type)
  • Need more performance?
  • FAQ
  • What are the differences between ${record["key"]} and ${key}?
  • Learn More

Was this helpful?

  1. Filter Plugins

record_transformer

The filter_record_transformer filter plugin mutates/transforms incoming event streams in a versatile manner. If there is a need to add/delete/modify events, this plugin is the first filter to try.

Example Configurations

filter_record_transformer is included in Fluentd's core. No installation required.

<filter foo.bar>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    tag ${tag}
  </record>
</filter>

The above filter adds the new field "hostname" with the server's hostname as its value (It is taking advantage of Ruby's string interpolation) and the new field "tag" with tag value. So, an input like

{"message":"hello world!"}

is transformed into

{"message":"hello world!", "hostname":"db001.internal.example.com", "tag":"foo.bar"}

Here is another example where the field "total" is divided by the field "count" to create a new field "avg":

<filter foo.bar>
  @type record_transformer
  enable_ruby
  <record>
    avg ${record["total"] / record["count"]}
  </record>
</filter>

It transforms an event like

{"total":100, "count":10}

into

{"total":100, "count":10, "avg":"10"}

With the enable_ruby option, an arbitrary Ruby expression can be used inside ${...}. Note that the "avg" field is typed as string in this example. You may use auto_typecast true option to treat the field as a float.

You can also use this plugin to modify your existing fields as

<filter foo.bar>
  @type record_transformer
  <record>
    message yay, ${record["message"]}
  </record>
</filter>

An input like

{"message":"hello world!"}

is transformed into

{"message":"yay, hello world!"}

Finally, this configuration embeds the value of the second part of the tag in the field "service_name". It might come in handy when aggregating data across many services.

<filter web.*>
  @type record_transformer
  <record>
    service_name ${tag_parts[1]}
  </record>
</filter>

So, if an event with the tag "web.auth" and record {"user_id":1, "status":"ok"} comes in, it transforms it into {"user_id":1, "status":"ok", "service_name":"auth"}.

Parameters

<record> directive

Parameters inside <record> directives are considered to be new key-value pairs:

<record>
  NEW_FIELD NEW_VALUE
</record>

For NEW_FIELD and NEW_VALUE, a special syntax ${} allows the user to generate a new field dynamically. Inside the curly braces, the following variables are available:

  • The incoming event's existing values can be referred by their field

    names. So, if the record is {"total":100, "count":10}, then

    record["total"]=100 and record["count"]=10.

  • tag refers to the whole tag.

  • time refers to stringanized event time.

  • hostname refers to machine's hostname. The actual value is result

    of

You can also access to a certain potion of a tag using the following notations:

  • tag_parts[N] refers to the Nth part of the tag.

  • tag_prefix[N] refers to the [0..N] part of the tag.

  • tag_suffix[N] refers to the [N..] part of the tag.

All indices are zero-based. For example, if you have an incoming event tagged debug.my.app, then tag_parts[1] will represent "my". Also in this case, tag_prefix[N] and tag_suffix[N] will work as follows:

tag_prefix[0] = debug          tag_suffix[0] = debug.my.app
tag_prefix[1] = debug.my       tag_suffix[1] = my.app
tag_prefix[2] = debug.my.app   tag_suffix[2] = app

enable_ruby (optional)

When set to true, the full Ruby syntax is enabled in the ${...} expression. The default value is false.

With true, additional variables could be used inside ${}.

  • record refers to the whole record.

  • time refers to event time as Time object, not stringanized event

    time.

Here is the examples:

jsonized_record ${record.to_json}
avg ${record["total"] / record["count"]}
formatted_time ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
escaped_tag ${tag.gsub('.', '-')}
last_tag ${tag_parts.last}
foo_${record["key"]} bar_${record["value"]}

auto_typecast (optional)

Automatically cast the field types. Default is false.

LIMITATION: This option is effective only for field values comprised of a single placeholder.

Effective Examples:

foo ${record["foo"]}

Non-Effective Examples:

foo ${record["foo"]}${record["bar"]}
foo ${record["foo"]}bar
foo 1

Internally, this keeps the original value type only when a single placeholder is used.

renew_record (optional)

By default, the record transformer filter mutates the incoming data. However, if this parameter is set to true, it modifies a new empty hash instead.

renew_time_key (optional, string type)

renew_time_key foo overwrites the time of events with a value of the record field foo if exists. The value of foo must be a unix time.

keep_keys (optional, array type)

A list of keys to keep. Only relevant if renew_record is set to true.

remove_keys (optional, array type)

A list of keys to delete.

Need more performance?

FAQ

What are the differences between ${record["key"]} and ${key}?

${key} is short-cut for ${record["key"]}. This is error prone because ${tag} is unclear for event tag or record["tag"]. So the ${key} syntax is now deprecated for avoiding this problem. Don't use ${key} short-cut syntax on the production.

Since v0.14, ${key} short-cut syntax is removed.

Learn More

PreviousFilter PluginsNextgrep

Last updated 5 years ago

Was this helpful?

.

is light-weight and faster version of filter_record_transformer. filter_record_modifier doesn't provide several filter_record_transformer features, but it covers popular cases. If you need better performace for mutating records, consider filter_record_modifier instead.

If this article is incorrect or outdated, or omits critical information, please . is a open source project under . All components are available under the Apache 2 License.

Socket.gethostname
filter_record_modifier
Filter Plugin Overview
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)