rewrite_tag_filter

The out_rewrite_tag_filter Output plugin has designed to rewrite tag like mod_rewrite. Re-emit a record with rewrited tag when a value matches/unmatches with the regular expression. Also you can change a tag from apache log by domain, status-code(ex. 500 error), user-agent, request-uri, regex-backreference and so on with regular expression.

How it works

It is a sample to arrange the tags by the regexp matched value of 'message'.

# Configuration
<match app.message>
  @type rewrite_tag_filter
  <rule>
    key message
    pattern ^\[(\w+)\] $1.${tag}
    tag $1.${tag}
  </rule>
</match>

:::text
+----------------------------------------+        +----------------------------------------------+
| original record                        |        | rewrited tag record                          |
|----------------------------------------|        |----------------------------------------------|
| app.message {"message":"[info]: ..."}  | +----> | info.app.message {"message":"[info]: ..."}   |
| app.message {"message":"[warn]: ..."}  | +----> | warn.app.message {"message":"[warn]: ..."}   |
| app.message {"message":"[crit]: ..."}  | +----> | crit.app.message {"message":"[crit]: ..."}   |
| app.message {"message":"[alert]: ..."} | +----> | alert.app.message {"message":"[alert]: ..."} |
+----------------------------------------+        +----------------------------------------------+

Install

out_rewrite_tag_filter is included in td-agent by default (v1.1.18 or later). Fluentd gem users will have to install the fluent-plugin-rewrite-tag-filter gem using the following command.

$ fluent-gem install fluent-plugin-rewrite-tag-filter

Example Configuration

Configuration design is dropping some pattern record first, then re-emit other matched record as new tag name.

<match apache.access>
  @type rewrite_tag_filter
  capitalize_regex_backreference yes
  <rule>
    key     path
    pattern \.(gif|jpe?g|png|pdf|zip)$
    tag     clear
  </rule>
  <rule>
    key     status
    pattern ^200$
    tag     clear
    invert  true
  </rule>
  <rule>
    key     domain
    pattern ^.+\.com$
    tag     clear
    invert  true
  </rule>
  <rule>
    key     domain
    pattern ^maps\.example\.com$
    tag     site.ExampleMaps
  </rule>
  <rule>
    key     domain
    pattern ^news\.example\.com$
    tag     site.ExampleNews
  </rule>
  # it is also supported regexp back reference.
  <rule>
    key     domain
    pattern ^(mail)\.(example)\.com$
    tag     site.$2$1
  </rule>
  <rule>
    key     domain
    pattern .+
    tag     site.unmatched
  </rule>
</match>

<match clear>
  @type null
</match>

Please see the README.md for further details.

Parameters

rewriteruleN (required at least one)

This is deprecated since 1.6.0. Use \ section.

rewriterule<num> <key> <regex_pattern> <new_tag>

It works with the order \ ascending, regexp matching \ for the values of \ from each record, re-emit with \.

capitalize_regex_backreference

Capitalize letter for every matched regex backreference. (ex: maps -> Maps)

hostname_command

Override hostname command for placeholder. (default setting is long hostname)

log_level option

The log_level option allows the user to set different levels of logging for each plugin. The supported log levels are: fatal, error, warn, info, debug, and trace.

Please see the logging article for further details.

<rule> section (optional) (multiple)

key (string) (required): The field name to which the regular
expression is applied
pattern (regexp) (required): The regular expression
tag (string) (required): New tag
invert (bool) (optional): If true, rewrite tag when unmatch
pattern
- Default value: false

It works with the order of appearance, regexp matching rule/pattern for the values of rule/key from each record, re-emit with rule/tag.

Placeholders

It is supported these placeholder for new_tag (rewrited tag). See more details at README.md

${tag}
__TAG__
{$tag_parts[n]}
__TAG_PARTS[n]__
${hostname}
__HOSTNAME__

Use cases

Aggregate + display 404 status pages by URL and referrer to find and
fix dead links.
Send an IRC alert for 5xx status codes on exceeding thresholds.

Aggregate + display 404 status pages by URL and referrer to find and fix dead links.

Collect access log from multiple application servers (config1)
Sum up the 404 error and output to mongoDB (config2)

Note: These plugins are required to be installed. * fluent-plugin-rewrite-tag-filter * fluent-plugin-mongo

[Config1] Application Servers

# Input access log to fluentd with embedded in_tail plugin
<source>
  @type tail
  path /var/log/httpd/access_log
  format apache2
  time_format %d/%b/%Y:%H:%M:%S %z
  tag apache.access
  pos_file /var/log/td-agent/apache_access.pos
</source>

# Forward to monitoring server
<match apache.access>
  @type forward
  flush_interval 5s
  <server>
    name server_name
    host 10.100.1.20
  </server>
</match>

[Config2] Monitoring Server

# built-in TCP input
<source>
  @type forward
</source>

# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
<match apache.access>
  @type rewrite_tag_filter
  <rule>
    key     status
    pattern ^(?!404)$
    tag     clear
  </rule>
  <rule>
    key      path
    pattern .+
    tag      mongo.apache.access.error404
  </rule>
</match>

# Store deadlinks log into mongoDB
<match mongo.apache.access.error404>
  @type        mongo
  host        10.100.1.30
  database    apache
  collection  deadlinks
  capped
  capped_size 50m
</match>

# Clear tag
<match clear>
  @type null
</match>

Send an IRC alert for 5xx status codes on exceeding thresholds.

Collect access log from multiple application servers (config1)
Sum up the 500 error and notify IRC and logging details to mongoDB
(config2)

Note: These plugins are required to be installed. * fluent-plugin-rewrite-tag-filter * fluent-plugin-datacounter * fluent-plugin-notifier * fluent-plugin-parser * fluent-plugin-mongo * fluent-plugin-irc

[Config1] Application Servers

# Input access log to fluentd with embedded in_tail plugin
# sample results: {"host":"127.0.0.1","user":null,"method":"GET","path":"/","code":500,"size":5039,"referer":null,"agent":"Mozilla"}
<source>
  @type tail
  path /var/log/httpd/access_log
  format apache2
  time_format %d/%b/%Y:%H:%M:%S %z
  tag apache.access
  pos_file /var/log/td-agent/apache_access.pos
</source>

# Forward to monitoring server
<match apache.access>
  @type forward
  flush_interval 5s
  <server>
    name server_name
    host 10.100.1.20
  </server>
</match>

[Config2] Monitoring Server

# built-in TCP input
<source>
  @type forward
</source>

# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
<match apache.access>
  @type copy
  <store>
    @type rewrite_tag_filter
    # drop static image record and redirect as 'count.apache.access'
    <rule>
      key     path
      pattern ^/(img|css|js|static|assets)/
      tag     clear
    </rule>
    <rule>
      key     path
      pattern .+
      tag     count.apache.access
    </rule>
  </store>
  <store>
    @type rewrite_tag_filter
    <rule>
      key     code
      pattern ^5\d\d$
      tag     mongo.apache.access.error5xx
    </rule>
  </store>
</match>

# Store 5xx error log into mongoDB
<match mongo.apache.access.error5xx>
  @type        mongo
  host        10.100.1.30
  database    apache
  collection  error_5xx
  capped
  capped_size 50m
</match>

# Count by status code
# sample results: {"unmatched_count":0,"unmatched_rate":0.0,"unmatched_percentage":0.0,"200_count":0,"200_rate":0.0,"200_percentage":0.0,"2xx_count":0,"2xx_rate":0.0,"2xx_percentage":0.0,"301_count":0,"301_rate":0.0,"301_percentage":0.0,"302_count":0,"302_rate":0.0,"302_percentage":0.0,"3xx_count":0,"3xx_rate":0.0,"3xx_percentage":0.0,"403_count":0,"403_rate":0.0,"403_percentage":0.0,"404_count":0,"404_rate":0.0,"404_percentage":0.0,"410_count":0,"410_rate":0.0,"410_percentage":0.0,"4xx_count":0,"4xx_rate":0.0,"4xx_percentage":0.0,"5xx_count":1,"5xx_rate":0.01,"5xx_percentage":100.0}
<match count.apache.access>
  @type datacounter
  unit minute
  outcast_unmatched false
  aggregate all
  tag threshold.apache.access
  count_key code
  pattern1 200 ^200$
  pattern2 2xx ^2\d\d$
  pattern3 301 ^301$
  pattern4 302 ^302$
  pattern5 3xx ^3\d\d$
  pattern6 403 ^403$
  pattern7 404 ^404$
  pattern8 410 ^410$
  pattern9 4xx ^4\d\d$
  pattern10 5xx ^5\d\d$
</match>

# Determine threshold
# sample results: {"pattern":"code_500","target_tag":"apache.access","target_key":"5xx_count","check_type":"numeric_upward","level":"warn","threshold":1.0,"value":1.0,"message_time":"2014-01-28 16:47:39 +0900"}
<match threshold.apache.access>
  @type notifier
  input_tag_remove_prefix threshold
  <def>
    pattern code_500
    check numeric_upward
    warn_threshold  10
    crit_threshold  40
    tag alert.http_5xx_error
    target_key_pattern ^5xx_count$
  </def>
</match>

# Generate message
# sample results: {"message":"HTTP Status warn [5xx_count] apache.access: 1.0 (threshold 1.0)"}
<match alert.http_5xx_error>
  @type deparser
  tag irc.http_5xx_error>
  format_key_names level,target_key,target_tag,value,threshold
  format HTTP Status %s [%s] %s: %s (threshold %s)
  key_name message
  reserve_data no
</match>

# Send IRC message
<match irc.http_5xx_error>
  @type irc
  host localhost
  port 6667
  channel fluentd
  nick fluentd
  user fluentd
  real fluentd
  message %s
  out_keys message
</match>

# Clear tag
<match clear>
  @type null
</match>

FAQ

With rewrite-tag-filter, logs are not forwarded. Why?

If you have following configuration, it doesn't work:

<match app.**>
  @type rewrite_tag_filter
  <rule>
    key     level
    pattern (.+)
    tag     app.$1
  </rule>
<match>

<match app.**>
  @type forward
  # ...
</match>

In this case, rewrite_tag_filter causes infinite loop because fluentd's routing is executed from top to bottom. So you need to change tag like below:

<match app.**>
  @type rewrite_tag_filter
  <rule>
    key     level
    pattern (.+)
    tag     level.app.$1
  </rule>
<match>

<match level.app.**>
  @type forward
  # ...
</match>

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.

Previousrelabel NextOthers

Last updated 5 years ago

Was this helpful?