rewrite_tag_filter

The out_rewrite_tag_filter Output plugin provides a rule-based mechanism for rewriting tags.

How it works

The plugin is configured by defining a list of rules containing conditional statements and information on how to rewrite matching tags.

When a message is handled by the plugin, the rules are tested one by one in order. If a matching rule is found, the message tag will be rewritten according to the definition in the rule and the message will be emitted again with the new tag.

Example

This in an example on how to use this plugin to re-write tags. In the example, records tagged with 'app.component' will have their tag prefixed with the value of the key 'message':

# Configuration
<match app.component>
@type rewrite_tag_filter
<rule>
key message
pattern /^\[(\w+)\]/
tag $1.${tag}
</rule>
</match>

Sample data:

+------------------------------------------+ +------------------------------------------------+
| original record | | rewritten tag record |
|------------------------------------------| |------------------------------------------------|
| app.component {"message":"[info]: ..."} | +----> | info.app.component {"message":"[info]: ..."} |
| app.component {"message":"[warn]: ..."} | +----> | warn.app.component {"message":"[warn]: ..."} |
| app.component {"message":"[crit]: ..."} | +----> | crit.app.component {"message":"[crit]: ..."} |
| app.component {"message":"[alert]: ..."} | +----> | alert.app.component {"message":"[alert]: ..."} |
+------------------------------------------+ +------------------------------------------------+

Installation

out_rewrite_tag_filter is included in td-agent by default (v3.0.1 or later). Fluentd gem users will have to install the fluent-plugin-rewrite-tag-filter gem using the following command.

$ fluent-gem install fluent-plugin-rewrite-tag-filter

For more details, see Plugin Management.

Configuration example

Configuration design is dropping some pattern record first, then re-emit other matched record as new tag name. The example configuration shown below gives an example on how the plugin can be used to define a number of rules that examines values from different keys and sets the tag depending on the regular expression configured in each rule.

The tag value is later used to decide whether the log event shall be dropped or not.

<match apache.access>
@type rewrite_tag_filter
capitalize_regex_backreference yes
<rule>
key path
pattern /\.(gif|jpe?g|png|pdf|zip)$/
tag clear
</rule>
<rule>
key status
pattern /^200$/
tag clear
invert true
</rule>
<rule>
key domain
pattern /^.+\.com$/
tag clear
invert true
</rule>
<rule>
key domain
pattern /^maps\.example\.com$/
tag site.ExampleMaps
</rule>
<rule>
key domain
pattern /^news\.example\.com$/
tag site.ExampleNews
</rule>
# it is also supported regexp back reference.
<rule>
key domain
pattern /^(mail)\.(example)\.com$/
tag site.$2$1
</rule>
<rule>
key domain
pattern /.+/
tag site.unmatched
</rule>
</match>
<match clear>
@type null
</match>

Please see the README.md for further details.

Parameters

rewriteruleN

This is obsoleted since 2.0.0. Use <rule> section.

capitalize_regex_backreference

type

default

version

bool

false

2.0.0

Capitalize letter for every matched regex backreference. (ex: maps -> Maps)

hostname_command

type

default

version

string

hostname

2.0.0

Override hostname command for placeholder. (default setting is long hostname)

<rule> section

It works with the order of appearance, regexp matching rule/pattern for the values of rule/key from each record, re-emit with rule/tag.

key

type

default

version

string

required parameter

2.0.0

The field name to which the regular expression is applied

pattern

type

default

version

regexp

required parameter

2.1.0

The regular expression which is applied on the field value

The type of pattern is string before 2.1.0.

tag

type

default

version

string

required parameter

2.0.0

New tag

invert** (bool) (optional):

type

default

version

bool

false

2.0.0

If true, rewrite tag when unmatch pattern

Placeholders

The following variable can be used when specifying the name of the rewritten tag. See more details at README.md.

  • ${tag}

  • __TAG__

  • ${tag_parts[n]}

  • __TAG_PARTS[n]__

  • ${hostname}

  • __HOSTNAME__

Use cases

  • Aggregate + display 404 status pages by URL and referrer to find and

    fix dead links.

  • Send an IRC alert for 5xx status codes on exceeding thresholds.

  • Collect access log from multiple application servers (config1)

  • Sum up the 404 error and output to mongoDB (config2)

Note: These plugins are required to be installed. * fluent-plugin-rewrite-tag-filter * fluent-plugin-mongo

[Config1] Application Servers

# Input access log to fluentd with embedded in_tail plugin
<source>
@type tail
path /var/log/httpd/access_log
<parse>
@type apache2
time_format %d/%b/%Y:%H:%M:%S %z
</parse>
tag apache.access
pos_file /var/log/td-agent/apache_access.pos
</source>
# Forward to monitoring server
<match apache.access>
@type forward
flush_interval 5s
<server>
name server_name
host 10.100.1.20
</server>
</match>

[Config2] Monitoring Server

# built-in TCP input
<source>
@type forward
</source>
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
<match apache.access>
@type rewrite_tag_filter
<rule>
key status
pattern /^(?!404)$/
tag clear
</rule>
<rule>
key path
pattern /.+/
tag mongo.apache.access.error404
</rule>
</match>
# Store deadlinks log into mongoDB
<match mongo.apache.access.error404>
@type mongo
host 10.100.1.30
database apache
collection deadlinks
capped
capped_size 50m
</match>
# Clear tag
<match clear>
@type null
</match>

Send an IRC alert for 5xx status codes on exceeding thresholds.

  • Collect access log from multiple application servers (config1)

  • Sum up the 500 error and notify IRC and logging details to mongoDB

    (config2)

Note: These plugins are required to be installed. * fluent-plugin-rewrite-tag-filter * fluent-plugin-datacounter * fluent-plugin-notifier * fluent-plugin-parser * fluent-plugin-mongo * fluent-plugin-irc

[Config1] Application Servers

# Input access log to fluentd with embedded in_tail plugin
# sample results: {"host":"127.0.0.1","user":null,"method":"GET","path":"/","code":500,"size":5039,"referer":null,"agent":"Mozilla"}
<source>
@type tail
path /var/log/httpd/access_log
<parse>
@type apache2
time_format %d/%b/%Y:%H:%M:%S %z
</parse>
tag apache.access
pos_file /var/log/td-agent/apache_access.pos
</source>
# Forward to monitoring server
<match apache.access>
@type forward
flush_interval 5s
<server>
name server_name
host 10.100.1.20
</server>
</match>

[Config2] Monitoring Server

# built-in TCP input
<source>
@type forward
</source>
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
<match apache.access>
@type copy
<store>
@type rewrite_tag_filter
# drop static image record and redirect as 'count.apache.access'
<rule>
key path
pattern /^\/(img|css|js|static|assets)\//
tag clear
</rule>
<rule>
key path
pattern /.+/
tag count.apache.access
</rule>
</store>
<store>
@type rewrite_tag_filter
<rule>
key code
pattern /^5\d\d$/
tag mongo.apache.access.error5xx
</rule>
</store>
</match>
# Store 5xx error log into mongoDB
<match mongo.apache.access.error5xx>
@type mongo
host 10.100.1.30
database apache
collection error_5xx
capped
capped_size 50m
</match>
# Count by status code
# sample results: {"unmatched_count":0,"unmatched_rate":0.0,"unmatched_percentage":0.0,"200_count":0,"200_rate":0.0,"200_percentage":0.0,"2xx_count":0,"2xx_rate":0.0,"2xx_percentage":0.0,"301_count":0,"301_rate":0.0,"301_percentage":0.0,"302_count":0,"302_rate":0.0,"302_percentage":0.0,"3xx_count":0,"3xx_rate":0.0,"3xx_percentage":0.0,"403_count":0,"403_rate":0.0,"403_percentage":0.0,"404_count":0,"404_rate":0.0,"404_percentage":0.0,"410_count":0,"410_rate":0.0,"410_percentage":0.0,"4xx_count":0,"4xx_rate":0.0,"4xx_percentage":0.0,"5xx_count":1,"5xx_rate":0.01,"5xx_percentage":100.0}
<match count.apache.access>
@type datacounter
unit minute
outcast_unmatched false
aggregate all
tag threshold.apache.access
count_key code
pattern1 200 ^200$
pattern2 2xx ^2\d\d$
pattern3 301 ^301$
pattern4 302 ^302$
pattern5 3xx ^3\d\d$
pattern6 403 ^403$
pattern7 404 ^404$
pattern8 410 ^410$
pattern9 4xx ^4\d\d$
pattern10 5xx ^5\d\d$
</match>
# Determine threshold
# sample results: {"pattern":"code_500","target_tag":"apache.access","target_key":"5xx_count","check_type":"numeric_upward","level":"warn","threshold":1.0,"value":1.0,"message_time":"2014-01-28 16:47:39 +0900"}
<match threshold.apache.access>
@type notifier
input_tag_remove_prefix threshold
<def>
pattern code_500
check numeric_upward
warn_threshold 10
crit_threshold 40
tag alert.http_5xx_error
target_key_pattern ^5xx_count$
</def>
</match>
# Generate message
# sample results: {"message":"HTTP Status warn [5xx_count] apache.access: 1.0 (threshold 1.0)"}
<match alert.http_5xx_error>
@type deparser
tag irc.http_5xx_error>
format_key_names level,target_key,target_tag,value,threshold
format HTTP Status %s [%s] %s: %s (threshold %s)
key_name message
reserve_data no
</match>
# Send IRC message
<match irc.http_5xx_error>
@type irc
host localhost
port 6667
channel fluentd
nick fluentd
user fluentd
real fluentd
message %s
out_keys message
</match>
# Clear tag
<match clear>
@type null
</match>

FAQ

With rewrite-tag-filter, logs are not forwarded. Why?

If you have following configuration, it doesn't work:

<match app.**>
@type rewrite_tag_filter
<rule>
key level
pattern /(.+)/
tag app.$1
</rule>
<match>
<match app.**>
@type forward
# ...
</match>

In this case, rewrite_tag_filter causes infinite loop because fluentd's routing is executed from top to bottom. So you need to change tag like below:

<match app.**>
@type rewrite_tag_filter
<rule>
key level
pattern /(.+)/
tag level.app.$1
</rule>
<match>
<match level.app.**>
@type forward
# ...
</match>

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.