rewrite_tag_filter

The out_rewrite_tag_filter Output plugin provides a rule-based mechanism for rewriting tags.
How it works
The plugin is configured by defining a list of rules containing conditional statements and information on how to rewrite matching tags.
When a message is handled by the plugin, the rules are tested one by one in order. If a matching rule is found, the message tag will be rewritten according to the definition in the rule and the message will be emitted again with the new tag.
Example
This in an example on how to use this plugin to re-write tags. In the example, records tagged with 'app.component' will have their tag prefixed with the value of the key 'message':
# Configuration
<match app.component>
  @type rewrite_tag_filter
  <rule>
    key message
    pattern /^\[(\w+)\]/
    tag $1.${tag}
  </rule>
</match>
Sample data:
+------------------------------------------+        +------------------------------------------------+
| original record                          |        | rewritten tag record                           |
|------------------------------------------|        |------------------------------------------------|
| app.component {"message":"[info]: ..."}  | +----> | info.app.component {"message":"[info]: ..."}   |
| app.component {"message":"[warn]: ..."}  | +----> | warn.app.component {"message":"[warn]: ..."}   |
| app.component {"message":"[crit]: ..."}  | +----> | crit.app.component {"message":"[crit]: ..."}   |
| app.component {"message":"[alert]: ..."} | +----> | alert.app.component {"message":"[alert]: ..."} |
+------------------------------------------+        +------------------------------------------------+
Installation
out_rewrite_tag_filter is included in td-agent by default (v3.0.1 or later). Fluentd gem users will have to install the fluent-plugin-rewrite-tag-filter gem using the following command.
$ fluent-gem install fluent-plugin-rewrite-tag-filter
For more details, see Plugin Management.
Configuration example
Configuration design is dropping some pattern record first, then re-emit other matched record as new tag name. The example configuration shown below gives an example on how the plugin can be used to define a number of rules that examines values from different keys and sets the tag depending on the regular expression configured in each rule.
The tag value is later used to decide whether the log event shall be dropped or not.
<match apache.access>
  @type rewrite_tag_filter
  capitalize_regex_backreference yes
  <rule>
    key     path
    pattern /\.(gif|jpe?g|png|pdf|zip)$/
    tag     clear
  </rule>
  <rule>
    key     status
    pattern /^200$/
    tag     clear
    invert  true
  </rule>
  <rule>
    key     domain
    pattern /^.+\.com$/
    tag     clear
    invert  true
  </rule>
  <rule>
    key     domain
    pattern /^maps\.example\.com$/
    tag     site.ExampleMaps
  </rule>
  <rule>
    key     domain
    pattern /^news\.example\.com$/
    tag     site.ExampleNews
  </rule>
  # it is also supported regexp back reference.
  <rule>
    key     domain
    pattern /^(mail)\.(example)\.com$/
    tag     site.$2$1
  </rule>
  <rule>
    key     domain
    pattern /.+/
    tag     site.unmatched
  </rule>
</match>
​
<match clear>
  @type null
</match>
Please see the README.md for further details.
Parameters
rewriteruleN
This is obsoleted since 2.0.0. Use <rule> section.
capitalize_regex_backreference
type
default
version
bool
false
2.0.0
Capitalize letter for every matched regex backreference. (ex: maps -> Maps)
hostname_command
type
default
version
string
hostname
2.0.0
Override hostname command for placeholder. (default setting is long hostname)
<rule> section
It works with the order of appearance, regexp matching rule/pattern for the values of rule/key from each record, re-emit with rule/tag.
key
type
default
version
string
required parameter
2.0.0
The field name to which the regular expression is applied
pattern
type
default
version
regexp
required parameter
2.1.0
The regular expression which is applied on the field value
The type of pattern is string before 2.1.0.
tag
type
default
version
string
required parameter
2.0.0
New tag
invert** (bool) (optional):
type
default
version
bool
false
2.0.0
If true, rewrite tag when unmatch pattern
Placeholders
The following variable can be used when specifying the name of the rewritten tag. See more details at README.md.
${tag}
__TAG__
${tag_parts[n]}
__TAG_PARTS[n]__
${hostname}
__HOSTNAME__
Use cases
Aggregate + display 404 status pages by URL and referrer to find and
fix dead links.
Send an IRC alert for 5xx status codes on exceeding thresholds.
Aggregate + display 404 status pages by URL and referrer to find and fix dead links.
Collect access log from multiple application servers (config1)
Sum up the 404 error and output to mongoDB (config2)
Note: These plugins are required to be installed. * fluent-plugin-rewrite-tag-filter * fluent-plugin-mongo
[Config1] Application Servers
# Input access log to fluentd with embedded in_tail plugin
<source>
  @type tail
  path /var/log/httpd/access_log
  <parse>
    @type apache2
    time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
  tag apache.access
  pos_file /var/log/td-agent/apache_access.pos
</source>
​
# Forward to monitoring server
<match apache.access>
  @type forward
  flush_interval 5s
  <server>
    name server_name
    host 10.100.1.20
  </server>
</match>
[Config2] Monitoring Server
# built-in TCP input
<source>
  @type forward
</source>
​
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
<match apache.access>
  @type rewrite_tag_filter
  <rule>
    key     status
    pattern /^(?!404)$/
    tag     clear
  </rule>
  <rule>
    key     path
    pattern /.+/
    tag     mongo.apache.access.error404
  </rule>
</match>
​
# Store deadlinks log into mongoDB
<match mongo.apache.access.error404>
  @type        mongo
  host        10.100.1.30
  database    apache
  collection  deadlinks
  capped
  capped_size 50m
</match>
​
# Clear tag
<match clear>
  @type null
</match>
Send an IRC alert for 5xx status codes on exceeding thresholds.
Collect access log from multiple application servers (config1)
Sum up the 500 error and notify IRC and logging details to mongoDB
(config2)
Note: These plugins are required to be installed. * fluent-plugin-rewrite-tag-filter * fluent-plugin-datacounter * fluent-plugin-notifier * fluent-plugin-parser * fluent-plugin-mongo * fluent-plugin-irc
[Config1] Application Servers
# Input access log to fluentd with embedded in_tail plugin
# sample results: {"host":"127.0.0.1","user":null,"method":"GET","path":"/","code":500,"size":5039,"referer":null,"agent":"Mozilla"}
<source>
  @type tail
  path /var/log/httpd/access_log
  <parse>
    @type apache2
    time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
  tag apache.access
  pos_file /var/log/td-agent/apache_access.pos
</source>
​
# Forward to monitoring server
<match apache.access>
  @type forward
  flush_interval 5s
  <server>
    name server_name
    host 10.100.1.20
  </server>
</match>
[Config2] Monitoring Server
# built-in TCP input
<source>
  @type forward
</source>
​
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
<match apache.access>
  @type copy
  <store>
    @type rewrite_tag_filter
    # drop static image record and redirect as 'count.apache.access'
    <rule>
      key     path
      pattern /^\/(img|css|js|static|assets)\//
      tag     clear
    </rule>
    <rule>
      key     path
      pattern /.+/
      tag     count.apache.access
    </rule>
  </store>
  <store>
    @type rewrite_tag_filter
    <rule>
      key     code
      pattern /^5\d\d$/
      tag     mongo.apache.access.error5xx
    </rule>
  </store>
</match>
​
# Store 5xx error log into mongoDB
<match mongo.apache.access.error5xx>
  @type        mongo
  host        10.100.1.30
  database    apache
  collection  error_5xx
  capped
  capped_size 50m
</match>
​
# Count by status code
# sample results: {"unmatched_count":0,"unmatched_rate":0.0,"unmatched_percentage":0.0,"200_count":0,"200_rate":0.0,"200_percentage":0.0,"2xx_count":0,"2xx_rate":0.0,"2xx_percentage":0.0,"301_count":0,"301_rate":0.0,"301_percentage":0.0,"302_count":0,"302_rate":0.0,"302_percentage":0.0,"3xx_count":0,"3xx_rate":0.0,"3xx_percentage":0.0,"403_count":0,"403_rate":0.0,"403_percentage":0.0,"404_count":0,"404_rate":0.0,"404_percentage":0.0,"410_count":0,"410_rate":0.0,"410_percentage":0.0,"4xx_count":0,"4xx_rate":0.0,"4xx_percentage":0.0,"5xx_count":1,"5xx_rate":0.01,"5xx_percentage":100.0}
<match count.apache.access>
  @type datacounter
  unit minute
  outcast_unmatched false
  aggregate all
  tag threshold.apache.access
  count_key code
  pattern1 200 ^200$
  pattern2 2xx ^2\d\d$
  pattern3 301 ^301$
  pattern4 302 ^302$
  pattern5 3xx ^3\d\d$
  pattern6 403 ^403$
  pattern7 404 ^404$
  pattern8 410 ^410$
  pattern9 4xx ^4\d\d$
  pattern10 5xx ^5\d\d$
</match>
​
# Determine threshold
# sample results: {"pattern":"code_500","target_tag":"apache.access","target_key":"5xx_count","check_type":"numeric_upward","level":"warn","threshold":1.0,"value":1.0,"message_time":"2014-01-28 16:47:39 +0900"}
<match threshold.apache.access>
  @type notifier
  input_tag_remove_prefix threshold
  <def>
    pattern code_500
    check numeric_upward
    warn_threshold  10
    crit_threshold  40
    tag alert.http_5xx_error
    target_key_pattern ^5xx_count$
  </def>
</match>
​
# Generate message
# sample results: {"message":"HTTP Status warn [5xx_count] apache.access: 1.0 (threshold 1.0)"}
<match alert.http_5xx_error>
  @type deparser
  tag irc.http_5xx_error>
  format_key_names level,target_key,target_tag,value,threshold
  format HTTP Status %s [%s] %s: %s (threshold %s)
  key_name message
  reserve_data no
</match>
​
# Send IRC message
<match irc.http_5xx_error>
  @type irc
  host localhost
  port 6667
  channel fluentd
  nick fluentd
  user fluentd
  real fluentd
  message %s
  out_keys message
</match>
​
# Clear tag
<match clear>
  @type null
</match>
FAQ
With rewrite-tag-filter, logs are not forwarded. Why?
If you have following configuration, it doesn't work:
<match app.**>
  @type rewrite_tag_filter
  <rule>
    key     level
    pattern /(.+)/
    tag     app.$1
  </rule>
</match>
​
<match app.**>
  @type forward
  # ...
</match>
In this case, rewrite_tag_filter causes infinite loop because fluentd's routing is executed from top to bottom. So you need to change tag like below:
<match app.**>
  @type rewrite_tag_filter
  <rule>
    key     level
    pattern /(.+)/
    tag     level.app.$1
  </rule>
</match>
​
<match level.app.**>
  @type forward
  # ...
</match>
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
mongo_replset
webhdfs
Last updated 2 weeks ago