rewrite_tag_filter

Search…
rewrite_tag_filter
The out_rewrite_tag_filter Output plugin provides a rule-based mechanism for rewriting tags.
How It Works
The plugin is configured by defining a list of rules containing conditional statements and information on how to rewrite the matching tags.
When a message is handled by the plugin, the rules are tested one by one in order. If a matching rule is found, the message tag will be rewritten according to the definition in the rule and the message will be emitted again with the new tag.
Example
Basic Example
This in an example of how to use this plugin to rewrite tags. In the example, records tagged with app.component will have their tag prefixed with the value of the key message:
1
<match app.component>
2
  @type rewrite_tag_filter
3
  <rule>
4
    key message
5
    pattern /^\[(\w+)\]/
6
    tag $1.${tag}
7
  </rule>
8
</match>
Copied!
Sample data:
1
+------------------------------------------+        +------------------------------------------------+
2
| original record                          |        | rewritten tag record                           |
3
|------------------------------------------|        |------------------------------------------------|
4
| app.component {"message":"[info]: ..."}  | +----> | info.app.component {"message":"[info]: ..."}   |
5
| app.component {"message":"[warn]: ..."}  | +----> | warn.app.component {"message":"[warn]: ..."}   |
6
| app.component {"message":"[crit]: ..."}  | +----> | crit.app.component {"message":"[crit]: ..."}   |
7
| app.component {"message":"[alert]: ..."} | +----> | alert.app.component {"message":"[alert]: ..."} |
8
+------------------------------------------+        +------------------------------------------------+
Copied!
Nested kubernetes namespace attributes based rules
This is an example of how to use this plugin to rewrite tags with nested attributes which are kubernetes metadata. In the example, records tagged with kubernetes.information will have their tag prefixed with the value of the nested key kubernetes.namespace_name.
Dot notation
1
<match kubernetes.**>
2
  @type rewrite_tag_filter
3
  <rule>
4
    key $.kubernetes.namespace_name
5
    pattern ^(.+)$
6
    tag $1.${tag}
7
  </rule>
8
</match>
Copied!
Bracket notation
1
<match kubernetes.**>
2
  @type rewrite_tag_filter
3
  <rule>
4
    key $['kubernetes']['namespace_name']
5
    pattern ^(.+)$
6
    tag $1.${tag}
7
  </rule>
8
</match>
Copied!
Sample data:
1
+----------------------------------------------------------------------------------------+        +-------------------------------------------------------------------------------------------------------+
2
| original record                                                                        |        | rewritten tag record                                                                                  |
3
|----------------------------------------------------------------------------------------|        |-------------------------------------------------------------------------------------------------------|
4
| kubernetes.information {"kubernetes" : { "namespace_name": "kube-system" }, ... }      | +----> | kube-system.kubernetes.information {"kubernetes" : { "namespace_name": "kube-system" }, ... }         |
5
| kubernetes.information {"kubernetes" : { "namespace_name": "default" }, ... }          | +----> | default.kubernetes.information {"kubernetes" : { "namespace_name": "default" }, ... }                 |
6
| kubernetes.information {"kubernetes" : { "namespace_name": "kube-public" }, ... }      | +----> | kube-public.kubernetes.information {"kubernetes" : { "namespace_name": "kube-public" }, ... }         |
7
| kubernetes.information {"kubernetes" : { "namespace_name": "kube-node-lease" }, ... }  | +----> | kube-node-lease.kubernetes.information {"kubernetes" : { "namespace_name": "kube-node-lease" }, ... } |
8
+----------------------------------------------------------------------------------------+        +-------------------------------------------------------------------------------------------------------+
Copied!
Installation
out_rewrite_tag_filter is included in td-agent by default (v3.0.1 or later). Fluentd gem users will have to install the fluent-plugin-rewrite-tag-filter gem using the following command:
1
$ fluent-gem install fluent-plugin-rewrite-tag-filter
Copied!
For more details, see Plugin Management.
Configuration Example
By design, the configuration drops some pattern records first and then it re-emits the next matched record as the new tag name. The example configuration shown below gives an example on how the plugin can be used to define a number of rules that examine values from different keys and sets the tag depending on the regular expression configured in each rule.
The tag value is later used to decide whether the log event shall be dropped or not.
1
<match apache.access>
2
  @type rewrite_tag_filter
3
  capitalize_regex_backreference yes
4
  <rule>
5
    key     path
6
    pattern /\.(gif|jpe?g|png|pdf|zip)$/
7
    tag     clear
8
  </rule>
9
  <rule>
10
    key     status
11
    pattern /^200$/
12
    tag     clear
13
    invert  true
14
  </rule>
15
  <rule>
16
    key     domain
17
    pattern /^.+\.com$/
18
    tag     clear
19
    invert  true
20
  </rule>
21
  <rule>
22
    key     domain
23
    pattern /^maps\.example\.com$/
24
    tag     site.ExampleMaps
25
  </rule>
26
  <rule>
27
    key     domain
28
    pattern /^news\.example\.com$/
29
    tag     site.ExampleNews
30
  </rule>
31
  # it is also supported regexp back reference.
32
  <rule>
33
    key     domain
34
    pattern /^(mail)\.(example)\.com$/
35
    tag     site.$2$1
36
  </rule>
37
  <rule>
38
    key     domain
39
    pattern /.+/
40
    tag     site.unmatched
41
  </rule>
42
</match>
43
​
44
<match clear>
45
  @type null
46
</match>
Copied!
Please see fluent-plugin-rewrite-tag-filter for further details.
Parameters
rewriteruleN
This is obsoleted since 2.0.0. Use <rule> section.
capitalize_regex_backreference
type
default
version
bool
false
2.0.0
Capitalizes letter for every matched regex backreference. (e.g. maps -> Maps)
hostname_command
type
default
version
string
hostname
2.0.0
Overrides hostname command for placeholder. (The default is the long hostname.)
<rule> Section
It works in the order of appearance, regexp matching rule/pattern for the values of rule/key from each record, re-emits with rule/tag.
key
type
default
version
string
required parameter
2.0.0
The field name to which the regular expression is applied.
pattern
type
default
version
regexp
required parameter
2.1.0
The regular expression which is applied on the field value.
The type of pattern is string before 2.1.0.
tag
type
default
version
string
required parameter
2.0.0
New tag.
invert** (bool) (optional):
type
default
version
bool
false
2.0.0
If true, rewrite tag when unmatch pattern.
Placeholders
The following variable can be used when specifying the name of the rewritten tag:
${tag}
__TAG__
${tag_parts[n]}
__TAG_PARTS[n]__
${hostname}
__HOSTNAME__
See more details at tag-placeholder.
Use Cases
Aggregate + display 404 status pages by URL and referrer to find and
fix dead links.
Send an IRC alert for 5xx status codes on exceeding thresholds.
Aggregate + display 404 status pages by URL and referrer to find and fix dead links.
Collect access log from multiple application servers (config1)
Sum up the 404 error and output to mongoDB (config2)
IMPORTANT
The plugins are required to be installed:
fluent-plugin-rewrite-tag-filter
fluent-plugin-mongo
[Config1] Application Servers
1
# Input access log to fluentd with embedded in_tail plugin
2
<source>
3
  @type tail
4
  path /var/log/httpd/access_log
5
  <parse>
6
    @type apache2
7
    time_format %d/%b/%Y:%H:%M:%S %z
8
  </parse>
9
  tag apache.access
10
  pos_file /var/log/td-agent/apache_access.pos
11
</source>
12
​
13
# Forward to monitoring server
14
<match apache.access>
15
  @type forward
16
  flush_interval 5s
17
  <server>
18
    name server_name
19
    host 10.100.1.20
20
  </server>
21
</match>
Copied!
[Config2] Monitoring Server
1
# built-in TCP input
2
<source>
3
  @type forward
4
</source>
5
​
6
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
7
<match apache.access>
8
  @type rewrite_tag_filter
9
  <rule>
10
    key     status
11
    pattern /^(?!404)$/
12
    tag     clear
13
  </rule>
14
  <rule>
15
    key     path
16
    pattern /.+/
17
    tag     mongo.apache.access.error404
18
  </rule>
19
</match>
20
​
21
# Store deadlinks log into mongoDB
22
<match mongo.apache.access.error404>
23
  @type        mongo
24
  host        10.100.1.30
25
  database    apache
26
  collection  deadlinks
27
  capped
28
  capped_size 50m
29
</match>
30
​
31
# Clear tag
32
<match clear>
33
  @type null
34
</match>
Copied!
Send an IRC alert for 5xx status codes on exceeding thresholds.
Collect access log from multiple application servers (config1)
Sum up the 500 error and notify IRC and logging details to mongoDB
(config2)
IMPORTANT
The plugins are required to be installed:
fluent-plugin-rewrite-tag-filter
fluent-plugin-mongo
fluent-plugin-datacounter
fluent-plugin-notifier
fluent-plugin-parser
fluent-plugin-irc
[Config1] Application Servers
1
# Input access log to fluentd with embedded in_tail plugin
2
# sample results: {"host":"127.0.0.1","user":null,"method":"GET","path":"/","code":500,"size":5039,"referer":null,"agent":"Mozilla"}
3
<source>
4
  @type tail
5
  path /var/log/httpd/access_log
6
  <parse>
7
    @type apache2
8
    time_format %d/%b/%Y:%H:%M:%S %z
9
  </parse>
10
  tag apache.access
11
  pos_file /var/log/td-agent/apache_access.pos
12
</source>
13
​
14
# Forward to monitoring server
15
<match apache.access>
16
  @type forward
17
  flush_interval 5s
18
  <server>
19
    name server_name
20
    host 10.100.1.20
21
  </server>
22
</match>
Copied!
[Config2] Monitoring Server
1
# built-in TCP input
2
<source>
3
  @type forward
4
</source>
5
​
6
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
7
<match apache.access>
8
  @type copy
9
  <store>
10
    @type rewrite_tag_filter
11
    # drop static image record and redirect as 'count.apache.access'
12
    <rule>
13
      key     path
14
      pattern /^\/(img|css|js|static|assets)\//
15
      tag     clear
16
    </rule>
17
    <rule>
18
      key     path
19
      pattern /.+/
20
      tag     count.apache.access
21
    </rule>
22
  </store>
23
  <store>
24
    @type rewrite_tag_filter
25
    <rule>
26
      key     code
27
      pattern /^5\d\d$/
28
      tag     mongo.apache.access.error5xx
29
    </rule>
30
  </store>
31
</match>
32
​
33
# Store 5xx error log into mongoDB
34
<match mongo.apache.access.error5xx>
35
  @type        mongo
36
  host        10.100.1.30
37
  database    apache
38
  collection  error_5xx
39
  capped
40
  capped_size 50m
41
</match>
42
​
43
# Count by status code
44
# sample results: {"unmatched_count":0,"unmatched_rate":0.0,"unmatched_percentage":0.0,"200_count":0,"200_rate":0.0,"200_percentage":0.0,"2xx_count":0,"2xx_rate":0.0,"2xx_percentage":0.0,"301_count":0,"301_rate":0.0,"301_percentage":0.0,"302_count":0,"302_rate":0.0,"302_percentage":0.0,"3xx_count":0,"3xx_rate":0.0,"3xx_percentage":0.0,"403_count":0,"403_rate":0.0,"403_percentage":0.0,"404_count":0,"404_rate":0.0,"404_percentage":0.0,"410_count":0,"410_rate":0.0,"410_percentage":0.0,"4xx_count":0,"4xx_rate":0.0,"4xx_percentage":0.0,"5xx_count":1,"5xx_rate":0.01,"5xx_percentage":100.0}
45
​
46
<match count.apache.access>
47
  @type datacounter
48
  unit minute
49
  outcast_unmatched false
50
  aggregate all
51
  tag threshold.apache.access
52
  count_key code
53
  pattern1 200 ^200$
54
  pattern2 2xx ^2\d\d$
55
  pattern3 301 ^301$
56
  pattern4 302 ^302$
57
  pattern5 3xx ^3\d\d$
58
  pattern6 403 ^403$
59
  pattern7 404 ^404$
60
  pattern8 410 ^410$
61
  pattern9 4xx ^4\d\d$
62
  pattern10 5xx ^5\d\d$
63
</match>
64
​
65
# Determine threshold
66
# sample results: {"pattern":"code_500","target_tag":"apache.access","target_key":"5xx_count","check_type":"numeric_upward","level":"warn","threshold":1.0,"value":1.0,"message_time":"2014-01-28 16:47:39 +0900"}
67
​
68
<match threshold.apache.access>
69
  @type notifier
70
  input_tag_remove_prefix threshold
71
  <def>
72
    pattern code_500
73
    check numeric_upward
74
    warn_threshold  10
75
    crit_threshold  40
76
    tag alert.http_5xx_error
77
    target_key_pattern ^5xx_count$
78
  </def>
79
</match>
80
​
81
# Generate message
82
# sample results: {"message":"HTTP Status warn [5xx_count] apache.access: 1.0 (threshold 1.0)"}
83
<match alert.http_5xx_error>
84
  @type deparser
85
  tag irc.http_5xx_error>
86
  format_key_names level,target_key,target_tag,value,threshold
87
  format HTTP Status %s [%s] %s: %s (threshold %s)
88
  key_name message
89
  reserve_data no
90
</match>
91
​
92
# Send IRC message
93
<match irc.http_5xx_error>
94
  @type irc
95
  host localhost
96
  port 6667
97
  channel fluentd
98
  nick fluentd
99
  user fluentd
100
  real fluentd
101
  message %s
102
  out_keys message
103
</match>
104
​
105
# Clear tag
106
<match clear>
107
  @type null
108
</match>
Copied!
FAQ
With rewrite-tag-filter, logs are not forwarded. Why?
If you have the following configuration, it doesn't work:
1
<match app.**>
2
  @type rewrite_tag_filter
3
  <rule>
4
    key     level
5
    pattern /(.+)/
6
    tag     app.$1
7
  </rule>
8
</match>
9
​
10
<match app.**>
11
  @type forward
12
  # ...
13
</match>
Copied!
In this case, rewrite_tag_filter causes an infinite loop because the fluentd's routing is executed from top-to-bottom. So, you need to change the tag like this:
1
<match app.**>
2
  @type rewrite_tag_filter
3
  <rule>
4
    key     level
5
    pattern /(.+)/
6
    tag     level.app.$1
7
  </rule>
8
</match>
9
​
10
<match level.app.**>
11
  @type forward
12
  # ...
13
</match>
Copied!
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.