Fluentd
Search…
rewrite_tag_filter
The out_rewrite_tag_filter Output plugin provides a rule-based mechanism for rewriting tags.

How It Works

The plugin is configured by defining a list of rules containing conditional statements and information on how to rewrite the matching tags.
When a message is handled by the plugin, the rules are tested one by one in order. If a matching rule is found, the message tag will be rewritten according to the definition in the rule and the message will be emitted again with the new tag.

Example

Basic Example

This in an example of how to use this plugin to rewrite tags. In the example, records tagged with app.component will have their tag prefixed with the value of the key message:
1
<match app.component>
2
@type rewrite_tag_filter
3
<rule>
4
key message
5
pattern /^\[(\w+)\]/
6
tag $1.${tag}
7
</rule>
8
</match>
Copied!
Sample data:
1
+------------------------------------------+ +------------------------------------------------+
2
| original record | | rewritten tag record |
3
|------------------------------------------| |------------------------------------------------|
4
| app.component {"message":"[info]: ..."} | +----> | info.app.component {"message":"[info]: ..."} |
5
| app.component {"message":"[warn]: ..."} | +----> | warn.app.component {"message":"[warn]: ..."} |
6
| app.component {"message":"[crit]: ..."} | +----> | crit.app.component {"message":"[crit]: ..."} |
7
| app.component {"message":"[alert]: ..."} | +----> | alert.app.component {"message":"[alert]: ..."} |
8
+------------------------------------------+ +------------------------------------------------+
Copied!

Nested kubernetes namespace attributes based rules

This is an example of how to use this plugin to rewrite tags with nested attributes which are kubernetes metadata. In the example, records tagged with kubernetes.information will have their tag prefixed with the value of the nested key kubernetes.namespace_name.
Dot notation
1
<match kubernetes.**>
2
@type rewrite_tag_filter
3
<rule>
4
key $.kubernetes.namespace_name
5
pattern ^(.+)$
6
tag $1.${tag}
7
</rule>
8
</match>
Copied!
Bracket notation
1
<match kubernetes.**>
2
@type rewrite_tag_filter
3
<rule>
4
key $['kubernetes']['namespace_name']
5
pattern ^(.+)$
6
tag $1.${tag}
7
</rule>
8
</match>
Copied!
Sample data:
1
+----------------------------------------------------------------------------------------+ +-------------------------------------------------------------------------------------------------------+
2
| original record | | rewritten tag record |
3
|----------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------------------------|
4
| kubernetes.information {"kubernetes" : { "namespace_name": "kube-system" }, ... } | +----> | kube-system.kubernetes.information {"kubernetes" : { "namespace_name": "kube-system" }, ... } |
5
| kubernetes.information {"kubernetes" : { "namespace_name": "default" }, ... } | +----> | default.kubernetes.information {"kubernetes" : { "namespace_name": "default" }, ... } |
6
| kubernetes.information {"kubernetes" : { "namespace_name": "kube-public" }, ... } | +----> | kube-public.kubernetes.information {"kubernetes" : { "namespace_name": "kube-public" }, ... } |
7
| kubernetes.information {"kubernetes" : { "namespace_name": "kube-node-lease" }, ... } | +----> | kube-node-lease.kubernetes.information {"kubernetes" : { "namespace_name": "kube-node-lease" }, ... } |
8
+----------------------------------------------------------------------------------------+ +-------------------------------------------------------------------------------------------------------+
Copied!

Installation

out_rewrite_tag_filter is included in td-agent by default (v3.0.1 or later). Fluentd gem users will have to install the fluent-plugin-rewrite-tag-filter gem using the following command:
1
$ fluent-gem install fluent-plugin-rewrite-tag-filter
Copied!
For more details, see Plugin Management.

Configuration Example

By design, the configuration drops some pattern records first and then it re-emits the next matched record as the new tag name. The example configuration shown below gives an example on how the plugin can be used to define a number of rules that examine values from different keys and sets the tag depending on the regular expression configured in each rule.
The tag value is later used to decide whether the log event shall be dropped or not.
1
<match apache.access>
2
@type rewrite_tag_filter
3
capitalize_regex_backreference yes
4
<rule>
5
key path
6
pattern /\.(gif|jpe?g|png|pdf|zip)$/
7
tag clear
8
</rule>
9
<rule>
10
key status
11
pattern /^200$/
12
tag clear
13
invert true
14
</rule>
15
<rule>
16
key domain
17
pattern /^.+\.com$/
18
tag clear
19
invert true
20
</rule>
21
<rule>
22
key domain
23
pattern /^maps\.example\.com$/
24
tag site.ExampleMaps
25
</rule>
26
<rule>
27
key domain
28
pattern /^news\.example\.com$/
29
tag site.ExampleNews
30
</rule>
31
# it is also supported regexp back reference.
32
<rule>
33
key domain
34
pattern /^(mail)\.(example)\.com$/
35
tag site.$2$1
36
</rule>
37
<rule>
38
key domain
39
pattern /.+/
40
tag site.unmatched
41
</rule>
42
</match>
43
44
<match clear>
45
@type null
46
</match>
Copied!
Please see fluent-plugin-rewrite-tag-filter for further details.

Parameters

rewriteruleN

This is obsoleted since 2.0.0. Use <rule> section.

capitalize_regex_backreference

type
default
version
bool
false
2.0.0
Capitalizes letter for every matched regex backreference. (e.g. maps -> Maps)

hostname_command

type
default
version
string
hostname
2.0.0
Overrides hostname command for placeholder. (The default is the long hostname.)

<rule> Section

It works in the order of appearance, regexp matching rule/pattern for the values of rule/key from each record, re-emits with rule/tag.

key

type
default
version
string
required parameter
2.0.0
The field name to which the regular expression is applied.

pattern

type
default
version
regexp
required parameter
2.1.0
The regular expression which is applied on the field value.
The type of pattern is string before 2.1.0.

tag

type
default
version
string
required parameter
2.0.0
New tag.

invert** (bool) (optional):

type
default
version
bool
false
2.0.0
If true, rewrite tag when unmatch pattern.

Placeholders

The following variable can be used when specifying the name of the rewritten tag:
  • ${tag}
  • __TAG__
  • ${tag_parts[n]}
  • __TAG_PARTS[n]__
  • ${hostname}
  • __HOSTNAME__
See more details at tag-placeholder.

Use Cases

  • Aggregate + display 404 status pages by URL and referrer to find and
    fix dead links.
  • Send an IRC alert for 5xx status codes on exceeding thresholds.

Aggregate + display 404 status pages by URL and referrer to find and fix dead links.

  • Collect access log from multiple application servers (config1)
  • Sum up the 404 error and output to mongoDB (config2)
IMPORTANT
The plugins are required to be installed:
  • fluent-plugin-rewrite-tag-filter
  • fluent-plugin-mongo
[Config1] Application Servers
1
# Input access log to fluentd with embedded in_tail plugin
2
<source>
3
@type tail
4
path /var/log/httpd/access_log
5
<parse>
6
@type apache2
7
time_format %d/%b/%Y:%H:%M:%S %z
8
</parse>
9
tag apache.access
10
pos_file /var/log/td-agent/apache_access.pos
11
</source>
12
13
# Forward to monitoring server
14
<match apache.access>
15
@type forward
16
flush_interval 5s
17
<server>
18
name server_name
19
host 10.100.1.20
20
</server>
21
</match>
Copied!
[Config2] Monitoring Server
1
# built-in TCP input
2
<source>
3
@type forward
4
</source>
5
6
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
7
<match apache.access>
8
@type rewrite_tag_filter
9
<rule>
10
key status
11
pattern /^(?!404)$/
12
tag clear
13
</rule>
14
<rule>
15
key path
16
pattern /.+/
17
tag mongo.apache.access.error404
18
</rule>
19
</match>
20
21
# Store deadlinks log into mongoDB
22
<match mongo.apache.access.error404>
23
@type mongo
24
host 10.100.1.30
25
database apache
26
collection deadlinks
27
capped
28
capped_size 50m
29
</match>
30
31
# Clear tag
32
<match clear>
33
@type null
34
</match>
Copied!

Send an IRC alert for 5xx status codes on exceeding thresholds.

  • Collect access log from multiple application servers (config1)
  • Sum up the 500 error and notify IRC and logging details to mongoDB
    (config2)
IMPORTANT
The plugins are required to be installed:
  • fluent-plugin-rewrite-tag-filter
  • fluent-plugin-mongo
  • fluent-plugin-datacounter
  • fluent-plugin-notifier
  • fluent-plugin-parser
  • fluent-plugin-irc
[Config1] Application Servers
1
# Input access log to fluentd with embedded in_tail plugin
2
# sample results: {"host":"127.0.0.1","user":null,"method":"GET","path":"/","code":500,"size":5039,"referer":null,"agent":"Mozilla"}
3
<source>
4
@type tail
5
path /var/log/httpd/access_log
6
<parse>
7
@type apache2
8
time_format %d/%b/%Y:%H:%M:%S %z
9
</parse>
10
tag apache.access
11
pos_file /var/log/td-agent/apache_access.pos
12
</source>
13
14
# Forward to monitoring server
15
<match apache.access>
16
@type forward
17
flush_interval 5s
18
<server>
19
name server_name
20
host 10.100.1.20
21
</server>
22
</match>
Copied!
[Config2] Monitoring Server
1
# built-in TCP input
2
<source>
3
@type forward
4
</source>
5
6
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
7
<match apache.access>
8
@type copy
9
<store>
10
@type rewrite_tag_filter
11
# drop static image record and redirect as 'count.apache.access'
12
<rule>
13
key path
14
pattern /^\/(img|css|js|static|assets)\//
15
tag clear
16
</rule>
17
<rule>
18
key path
19
pattern /.+/
20
tag count.apache.access
21
</rule>
22
</store>
23
<store>
24
@type rewrite_tag_filter
25
<rule>
26
key code
27
pattern /^5\d\d$/
28
tag mongo.apache.access.error5xx
29
</rule>
30
</store>
31
</match>
32
33
# Store 5xx error log into mongoDB
34
<match mongo.apache.access.error5xx>
35
@type mongo
36
host 10.100.1.30
37
database apache
38
collection error_5xx
39
capped
40
capped_size 50m
41
</match>
42
43
# Count by status code
44
# sample results: {"unmatched_count":0,"unmatched_rate":0.0,"unmatched_percentage":0.0,"200_count":0,"200_rate":0.0,"200_percentage":0.0,"2xx_count":0,"2xx_rate":0.0,"2xx_percentage":0.0,"301_count":0,"301_rate":0.0,"301_percentage":0.0,"302_count":0,"302_rate":0.0,"302_percentage":0.0,"3xx_count":0,"3xx_rate":0.0,"3xx_percentage":0.0,"403_count":0,"403_rate":0.0,"403_percentage":0.0,"404_count":0,"404_rate":0.0,"404_percentage":0.0,"410_count":0,"410_rate":0.0,"410_percentage":0.0,"4xx_count":0,"4xx_rate":0.0,"4xx_percentage":0.0,"5xx_count":1,"5xx_rate":0.01,"5xx_percentage":100.0}
45
46
<match count.apache.access>
47
@type datacounter
48
unit minute
49
outcast_unmatched false
50
aggregate all
51
tag threshold.apache.access
52
count_key code
53
pattern1 200 ^200$
54
pattern2 2xx ^2\d\d$
55
pattern3 301 ^301$
56
pattern4 302 ^302$
57
pattern5 3xx ^3\d\d$
58
pattern6 403 ^403$
59
pattern7 404 ^404$
60
pattern8 410 ^410$
61
pattern9 4xx ^4\d\d$
62
pattern10 5xx ^5\d\d$
63
</match>
64
65
# Determine threshold
66
# sample results: {"pattern":"code_500","target_tag":"apache.access","target_key":"5xx_count","check_type":"numeric_upward","level":"warn","threshold":1.0,"value":1.0,"message_time":"2014-01-28 16:47:39 +0900"}
67
68
<match threshold.apache.access>
69
@type notifier
70
input_tag_remove_prefix threshold
71
<def>
72
pattern code_500
73
check numeric_upward
74
warn_threshold 10
75
crit_threshold 40
76
tag alert.http_5xx_error
77
target_key_pattern ^5xx_count$
78
</def>
79
</match>
80
81
# Generate message
82
# sample results: {"message":"HTTP Status warn [5xx_count] apache.access: 1.0 (threshold 1.0)"}
83
<match alert.http_5xx_error>
84
@type deparser
85
tag irc.http_5xx_error>
86
format_key_names level,target_key,target_tag,value,threshold
87
format HTTP Status %s [%s] %s: %s (threshold %s)
88
key_name message
89
reserve_data no
90
</match>
91
92
# Send IRC message
93
<match irc.http_5xx_error>
94
@type irc
95
host localhost
96
port 6667
97
channel fluentd
98
nick fluentd
99
user fluentd
100
real fluentd
101
message %s
102
out_keys message
103
</match>
104
105
# Clear tag
106
<match clear>
107
@type null
108
</match>
Copied!

FAQ

With rewrite-tag-filter, logs are not forwarded. Why?

If you have the following configuration, it doesn't work:
1
<match app.**>
2
@type rewrite_tag_filter
3
<rule>
4
key level
5
pattern /(.+)/
6
tag app.$1
7
</rule>
8
</match>
9
10
<match app.**>
11
@type forward
12
# ...
13
</match>
Copied!
In this case, rewrite_tag_filter causes an infinite loop because the fluentd's routing is executed from top-to-bottom. So, you need to change the tag like this:
1
<match app.**>
2
@type rewrite_tag_filter
3
<rule>
4
key level
5
pattern /(.+)/
6
tag level.app.$1
7
</rule>
8
</match>
9
10
<match level.app.**>
11
@type forward
12
# ...
13
</match>
Copied!
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 6mo ago