rewrite_tag_filter

The out_rewrite_tag_filter Output plugin provides a rule-based mechanism for rewriting tags.

How It Works

The plugin is configured by defining a list of rules containing conditional statements and information on how to rewrite the matching tags.

When a message is handled by the plugin, the rules are tested one by one in order. If a matching rule is found, the message tag will be rewritten according to the definition in the rule and the message will be emitted again with the new tag.

Example

This in an example of how to use this plugin to rewrite tags. In the example, records tagged with app.component will have their tag prefixed with the value of the key message:

<match app.component>
@type rewrite_tag_filter
<rule>
key message
pattern /^(w+)$/
tag $1.${tag}
</rule>
</match>

Sample data:

+------------------------------------------+ +------------------------------------------------+
| original record | | rewritten tag record |
|------------------------------------------| |------------------------------------------------|
| app.component {"message":"[info]: ..."} | +----> | info.app.component {"message":"[info]: ..."} |
| app.component {"message":"[warn]: ..."} | +----> | warn.app.component {"message":"[warn]: ..."} |
| app.component {"message":"[crit]: ..."} | +----> | crit.app.component {"message":"[crit]: ..."} |
| app.component {"message":"[alert]: ..."} | +----> | alert.app.component {"message":"[alert]: ..."} |
+------------------------------------------+ +------------------------------------------------+

Installation

out_rewrite_tag_filter is included in td-agent by default (v3.0.1 or later). Fluentd gem users will have to install the fluent-plugin-rewrite-tag-filter gem using the following command:

$ fluent-gem install fluent-plugin-rewrite-tag-filter

For more details, see Plugin Management.

Configuration Example

Configuration design is dropping some pattern record first, then re-emit other matched record as new tag name. The example configuration shown below gives an example on how the plugin can be used to define a number of rules that examine values from different keys and sets the tag depending on the regular expression configured in each rule.

The tag value is later used to decide whether the log event shall be dropped or not.

<match apache.access>
@type rewrite_tag_filter
capitalize_regex_backreference yes
<rule>
key path
pattern /\.(gif|jpe?g|png|pdf|zip)$/
tag clear
</rule>
<rule>
key status
pattern /^200$/
tag clear
invert true
</rule>
<rule>
key domain
pattern /^.+\.com$/
tag clear
invert true
</rule>
<rule>
key domain
pattern /^maps\.example\.com$/
tag site.ExampleMaps
</rule>
<rule>
key domain
pattern /^news\.example\.com$/
tag site.ExampleNews
</rule>
# it is also supported regexp back reference.
<rule>
key domain
pattern /^(mail)\.(example)\.com$/
tag site.$2$1
</rule>
<rule>
key domain
pattern /.+/
tag site.unmatched
</rule>
</match>
<match clear>
@type null
</match>

Please see fluent-plugin-rewrite-tag-filter for further details.

Parameters

rewriteruleN

This is obsoleted since 2.0.0. Use <rule> section.

capitalize_regex_backreference

type

default

version

bool

false

2.0.0

Capitalizes letter for every matched regex backreference. (e.g. maps -> Maps)

hostname_command

type

default

version

string

hostname

2.0.0

Overrides hostname command for placeholder. (default setting is long hostname)

<rule> Section

It works in the order of appearance, regexp matching rule/pattern for the values of rule/key from each record, re-emits with rule/tag.

key

type

default

version

string

required parameter

2.0.0

The field name to which the regular expression is applied.

pattern

type

default

version

regexp

required parameter

2.1.0

The regular expression which is applied on the field value.

The type of pattern is string before 2.1.0.

tag

type

default

version

string

required parameter

2.0.0

New tag.

invert** (bool) (optional):

type

default

version

bool

false

2.0.0

If true, rewrite tag when unmatch pattern.

Placeholders

The following variable can be used when specifying the name of the rewritten tag:

  • ${tag}

  • __TAG__

  • ${tag_parts[n]}

  • __TAG_PARTS[n]__

  • ${hostname}

  • __HOSTNAME__

See more details at tag-placeholder.

Use Cases

  • Aggregate + display 404 status pages by URL and referrer to find and

    fix dead links.

  • Send an IRC alert for 5xx status codes on exceeding thresholds.

  • Collect access log from multiple application servers (config1)

  • Sum up the 404 error and output to mongoDB (config2)

IMPORTANT

The plugins are required to be installed:

  • fluent-plugin-rewrite-tag-filter

  • fluent-plugin-mongo

[Config1] Application Servers

# Input access log to fluentd with embedded in_tail plugin
<source>
@type tail
path /var/log/httpd/access_log
<parse>
@type apache2
time_format %d/%b/%Y:%H:%M:%S %z
</parse>
tag apache.access
pos_file /var/log/td-agent/apache_access.pos
</source>
# Forward to monitoring server
<match apache.access>
@type forward
flush_interval 5s
<server>
name server_name
host 10.100.1.20
</server>
</match>

[Config2] Monitoring Server

# built-in TCP input
<source>
@type forward
</source>
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
<match apache.access>
@type rewrite_tag_filter
<rule>
key status
pattern /^(?!404)$/
tag clear
</rule>
<rule>
key path
pattern /.+/
tag mongo.apache.access.error404
</rule>
</match>
# Store deadlinks log into mongoDB
<match mongo.apache.access.error404>
@type mongo
host 10.100.1.30
database apache
collection deadlinks
capped
capped_size 50m
</match>
# Clear tag
<match clear>
@type null
</match>

Send an IRC alert for 5xx status codes on exceeding thresholds.

  • Collect access log from multiple application servers (config1)

  • Sum up the 500 error and notify IRC and logging details to mongoDB

    (config2)

IMPORTANT

The plugins are required to be installed:

  • fluent-plugin-rewrite-tag-filter

  • fluent-plugin-mongo

  • fluent-plugin-datacounter

  • fluent-plugin-notifier

  • fluent-plugin-parser

  • fluent-plugin-irc

[Config1] Application Servers

# Input access log to fluentd with embedded in_tail plugin
# sample results: {"host":"127.0.0.1","user":null,"method":"GET","path":"/","code":500,"size":5039,"referer":null,"agent":"Mozilla"}
<source>
@type tail
path /var/log/httpd/access_log
<parse>
@type apache2
time_format %d/%b/%Y:%H:%M:%S %z
</parse>
tag apache.access
pos_file /var/log/td-agent/apache_access.pos
</source>
# Forward to monitoring server
<match apache.access>
@type forward
flush_interval 5s
<server>
name server_name
host 10.100.1.20
</server>
</match>

[Config2] Monitoring Server

# built-in TCP input
<source>
@type forward
</source>
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
<match apache.access>
@type copy
<store>
@type rewrite_tag_filter
# drop static image record and redirect as 'count.apache.access'
<rule>
key path
pattern /^\/(img|css|js|static|assets)\//
tag clear
</rule>
<rule>
key path
pattern /.+/
tag count.apache.access
</rule>
</store>
<store>
@type rewrite_tag_filter
<rule>
key code
pattern /^5\d\d$/
tag mongo.apache.access.error5xx
</rule>
</store>
</match>
# Store 5xx error log into mongoDB
<match mongo.apache.access.error5xx>
@type mongo
host 10.100.1.30
database apache
collection error_5xx
capped
capped_size 50m
</match>
# Count by status code
# sample results: {"unmatched_count":0,"unmatched_rate":0.0,"unmatched_percentage":0.0,"200_count":0,"200_rate":0.0,"200_percentage":0.0,"2xx_count":0,"2xx_rate":0.0,"2xx_percentage":0.0,"301_count":0,"301_rate":0.0,"301_percentage":0.0,"302_count":0,"302_rate":0.0,"302_percentage":0.0,"3xx_count":0,"3xx_rate":0.0,"3xx_percentage":0.0,"403_count":0,"403_rate":0.0,"403_percentage":0.0,"404_count":0,"404_rate":0.0,"404_percentage":0.0,"410_count":0,"410_rate":0.0,"410_percentage":0.0,"4xx_count":0,"4xx_rate":0.0,"4xx_percentage":0.0,"5xx_count":1,"5xx_rate":0.01,"5xx_percentage":100.0}
<match count.apache.access>
@type datacounter
unit minute
outcast_unmatched false
aggregate all
tag threshold.apache.access
count_key code
pattern1 200 ^200$
pattern2 2xx ^2\d\d$
pattern3 301 ^301$
pattern4 302 ^302$
pattern5 3xx ^3\d\d$
pattern6 403 ^403$
pattern7 404 ^404$
pattern8 410 ^410$
pattern9 4xx ^4\d\d$
pattern10 5xx ^5\d\d$
</match>
# Determine threshold
# sample results: {"pattern":"code_500","target_tag":"apache.access","target_key":"5xx_count","check_type":"numeric_upward","level":"warn","threshold":1.0,"value":1.0,"message_time":"2014-01-28 16:47:39 +0900"}
<match threshold.apache.access>
@type notifier
input_tag_remove_prefix threshold
<def>
pattern code_500
check numeric_upward
warn_threshold 10
crit_threshold 40
tag alert.http_5xx_error
target_key_pattern ^5xx_count$
</def>
</match>
# Generate message
# sample results: {"message":"HTTP Status warn [5xx_count] apache.access: 1.0 (threshold 1.0)"}
<match alert.http_5xx_error>
@type deparser
tag irc.http_5xx_error>
format_key_names level,target_key,target_tag,value,threshold
format HTTP Status %s [%s] %s: %s (threshold %s)
key_name message
reserve_data no
</match>
# Send IRC message
<match irc.http_5xx_error>
@type irc
host localhost
port 6667
channel fluentd
nick fluentd
user fluentd
real fluentd
message %s
out_keys message
</match>
# Clear tag
<match clear>
@type null
</match>

FAQ

With rewrite-tag-filter, logs are not forwarded. Why?

If you have the following configuration, it doesn't work:

<match app.**>
@type rewrite_tag_filter
<rule>
key level
pattern /(.+)/
tag app.$1
</rule>
</match>
<match app.**>
@type forward
# ...
</match>

In this case, rewrite_tag_filter causes infinite loop because fluentd's routing is executed from top to bottom. So you need to change tag like this:

<match app.**>
@type rewrite_tag_filter
<rule>
key level
pattern /(.+)/
tag level.app.$1
</rule>
</match>
<match level.app.**>
@type forward
# ...
</match>

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.