Fluentd
Search…
rewrite_tag_filter
The out_rewrite_tag_filter Output plugin has designed to rewrite tag like mod_rewrite. Re-emit a record with rewrited tag when a value matches/unmatches with the regular expression. Also you can change a tag from apache log by domain, status-code(ex. 500 error), user-agent, request-uri, regex-backreference and so on with regular expression.

How it works

It is a sample to arrange the tags by the regexp matched value of 'message'.
1
# Configuration
2
<match app.message>
3
@type rewrite_tag_filter
4
<rule>
5
key message
6
pattern ^\[(\w+)\] $1.${tag}
7
tag $1.${tag}
8
</rule>
9
</match>
10
11
:::text
12
+----------------------------------------+ +----------------------------------------------+
13
| original record | | rewrited tag record |
14
|----------------------------------------| |----------------------------------------------|
15
| app.message {"message":"[info]: ..."} | +----> | info.app.message {"message":"[info]: ..."} |
16
| app.message {"message":"[warn]: ..."} | +----> | warn.app.message {"message":"[warn]: ..."} |
17
| app.message {"message":"[crit]: ..."} | +----> | crit.app.message {"message":"[crit]: ..."} |
18
| app.message {"message":"[alert]: ..."} | +----> | alert.app.message {"message":"[alert]: ..."} |
19
+----------------------------------------+ +----------------------------------------------+
Copied!

Install

out_rewrite_tag_filter is included in td-agent by default (v1.1.18 or later). Fluentd gem users will have to install the fluent-plugin-rewrite-tag-filter gem using the following command.
1
$ fluent-gem install fluent-plugin-rewrite-tag-filter
Copied!

Example Configuration

Configuration design is dropping some pattern record first, then re-emit other matched record as new tag name.
1
<match apache.access>
2
@type rewrite_tag_filter
3
capitalize_regex_backreference yes
4
<rule>
5
key path
6
pattern \.(gif|jpe?g|png|pdf|zip)$
7
tag clear
8
</rule>
9
<rule>
10
key status
11
pattern ^200$
12
tag clear
13
invert true
14
</rule>
15
<rule>
16
key domain
17
pattern ^.+\.com$
18
tag clear
19
invert true
20
</rule>
21
<rule>
22
key domain
23
pattern ^maps\.example\.com$
24
tag site.ExampleMaps
25
</rule>
26
<rule>
27
key domain
28
pattern ^news\.example\.com$
29
tag site.ExampleNews
30
</rule>
31
# it is also supported regexp back reference.
32
<rule>
33
key domain
34
pattern ^(mail)\.(example)\.com$
35
tag site.$2$1
36
</rule>
37
<rule>
38
key domain
39
pattern .+
40
tag site.unmatched
41
</rule>
42
</match>
43
44
<match clear>
45
@type null
46
</match>
Copied!
Please see the README.md for further details.

Parameters

rewriteruleN (required at least one)

This is deprecated since 1.6.0. Use \ section.
rewriterule<num> <key> <regex_pattern> <new_tag>
It works with the order \ ascending, regexp matching \ for the values of \ from each record, re-emit with \.

capitalize_regex_backreference

Capitalize letter for every matched regex backreference. (ex: maps -> Maps)

hostname_command

Override hostname command for placeholder. (default setting is long hostname)

log_level option

The log_level option allows the user to set different levels of logging for each plugin. The supported log levels are: fatal, error, warn, info, debug, and trace.
Please see the logging article for further details.

<rule> section (optional) (multiple)

  • key (string) (required): The field name to which the regular
    expression is applied
  • pattern (regexp) (required): The regular expression
  • tag (string) (required): New tag
  • invert (bool) (optional): If true, rewrite tag when unmatch
    pattern
    • Default value: false
It works with the order of appearance, regexp matching rule/pattern for the values of rule/key from each record, re-emit with rule/tag.

Placeholders

It is supported these placeholder for new_tag (rewrited tag). See more details at README.md
  • ${tag}
  • __TAG__
  • {$tag_parts[n]}
  • __TAG_PARTS[n]__
  • ${hostname}
  • __HOSTNAME__

Use cases

  • Aggregate + display 404 status pages by URL and referrer to find and
    fix dead links.
  • Send an IRC alert for 5xx status codes on exceeding thresholds.

Aggregate + display 404 status pages by URL and referrer to find and fix dead links.

  • Collect access log from multiple application servers (config1)
  • Sum up the 404 error and output to mongoDB (config2)
Note: These plugins are required to be installed. * fluent-plugin-rewrite-tag-filter * fluent-plugin-mongo
[Config1] Application Servers
1
# Input access log to fluentd with embedded in_tail plugin
2
<source>
3
@type tail
4
path /var/log/httpd/access_log
5
format apache2
6
time_format %d/%b/%Y:%H:%M:%S %z
7
tag apache.access
8
pos_file /var/log/td-agent/apache_access.pos
9
</source>
10
11
# Forward to monitoring server
12
<match apache.access>
13
@type forward
14
flush_interval 5s
15
<server>
16
name server_name
17
host 10.100.1.20
18
</server>
19
</match>
Copied!
[Config2] Monitoring Server
1
# built-in TCP input
2
<source>
3
@type forward
4
</source>
5
6
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
7
<match apache.access>
8
@type rewrite_tag_filter
9
<rule>
10
key status
11
pattern ^(?!404)$
12
tag clear
13
</rule>
14
<rule>
15
key path
16
pattern .+
17
tag mongo.apache.access.error404
18
</rule>
19
</match>
20
21
# Store deadlinks log into mongoDB
22
<match mongo.apache.access.error404>
23
@type mongo
24
host 10.100.1.30
25
database apache
26
collection deadlinks
27
capped
28
capped_size 50m
29
</match>
30
31
# Clear tag
32
<match clear>
33
@type null
34
</match>
Copied!

Send an IRC alert for 5xx status codes on exceeding thresholds.

  • Collect access log from multiple application servers (config1)
  • Sum up the 500 error and notify IRC and logging details to mongoDB
    (config2)
Note: These plugins are required to be installed. * fluent-plugin-rewrite-tag-filter * fluent-plugin-datacounter * fluent-plugin-notifier * fluent-plugin-parser * fluent-plugin-mongo * fluent-plugin-irc
[Config1] Application Servers
1
# Input access log to fluentd with embedded in_tail plugin
2
# sample results: {"host":"127.0.0.1","user":null,"method":"GET","path":"/","code":500,"size":5039,"referer":null,"agent":"Mozilla"}
3
<source>
4
@type tail
5
path /var/log/httpd/access_log
6
format apache2
7
time_format %d/%b/%Y:%H:%M:%S %z
8
tag apache.access
9
pos_file /var/log/td-agent/apache_access.pos
10
</source>
11
12
# Forward to monitoring server
13
<match apache.access>
14
@type forward
15
flush_interval 5s
16
<server>
17
name server_name
18
host 10.100.1.20
19
</server>
20
</match>
Copied!
[Config2] Monitoring Server
1
# built-in TCP input
2
<source>
3
@type forward
4
</source>
5
6
# Filter record like mod_rewrite with fluent-plugin-rewrite-tag-filter
7
<match apache.access>
8
@type copy
9
<store>
10
@type rewrite_tag_filter
11
# drop static image record and redirect as 'count.apache.access'
12
<rule>
13
key path
14
pattern ^/(img|css|js|static|assets)/
15
tag clear
16
</rule>
17
<rule>
18
key path
19
pattern .+
20
tag count.apache.access
21
</rule>
22
</store>
23
<store>
24
@type rewrite_tag_filter
25
<rule>
26
key code
27
pattern ^5\d\d$
28
tag mongo.apache.access.error5xx
29
</rule>
30
</store>
31
</match>
32
33
# Store 5xx error log into mongoDB
34
<match mongo.apache.access.error5xx>
35
@type mongo
36
host 10.100.1.30
37
database apache
38
collection error_5xx
39
capped
40
capped_size 50m
41
</match>
42
43
# Count by status code
44
# sample results: {"unmatched_count":0,"unmatched_rate":0.0,"unmatched_percentage":0.0,"200_count":0,"200_rate":0.0,"200_percentage":0.0,"2xx_count":0,"2xx_rate":0.0,"2xx_percentage":0.0,"301_count":0,"301_rate":0.0,"301_percentage":0.0,"302_count":0,"302_rate":0.0,"302_percentage":0.0,"3xx_count":0,"3xx_rate":0.0,"3xx_percentage":0.0,"403_count":0,"403_rate":0.0,"403_percentage":0.0,"404_count":0,"404_rate":0.0,"404_percentage":0.0,"410_count":0,"410_rate":0.0,"410_percentage":0.0,"4xx_count":0,"4xx_rate":0.0,"4xx_percentage":0.0,"5xx_count":1,"5xx_rate":0.01,"5xx_percentage":100.0}
45
<match count.apache.access>
46
@type datacounter
47
unit minute
48
outcast_unmatched false
49
aggregate all
50
tag threshold.apache.access
51
count_key code
52
pattern1 200 ^200$
53
pattern2 2xx ^2\d\d$
54
pattern3 301 ^301$
55
pattern4 302 ^302$
56
pattern5 3xx ^3\d\d$
57
pattern6 403 ^403$
58
pattern7 404 ^404$
59
pattern8 410 ^410$
60
pattern9 4xx ^4\d\d$
61
pattern10 5xx ^5\d\d$
62
</match>
63
64
# Determine threshold
65
# sample results: {"pattern":"code_500","target_tag":"apache.access","target_key":"5xx_count","check_type":"numeric_upward","level":"warn","threshold":1.0,"value":1.0,"message_time":"2014-01-28 16:47:39 +0900"}
66
<match threshold.apache.access>
67
@type notifier
68
input_tag_remove_prefix threshold
69
<def>
70
pattern code_500
71
check numeric_upward
72
warn_threshold 10
73
crit_threshold 40
74
tag alert.http_5xx_error
75
target_key_pattern ^5xx_count$
76
</def>
77
</match>
78
79
# Generate message
80
# sample results: {"message":"HTTP Status warn [5xx_count] apache.access: 1.0 (threshold 1.0)"}
81
<match alert.http_5xx_error>
82
@type deparser
83
tag irc.http_5xx_error>
84
format_key_names level,target_key,target_tag,value,threshold
85
format HTTP Status %s [%s] %s: %s (threshold %s)
86
key_name message
87
reserve_data no
88
</match>
89
90
# Send IRC message
91
<match irc.http_5xx_error>
92
@type irc
93
host localhost
94
port 6667
95
channel fluentd
96
nick fluentd
97
user fluentd
98
real fluentd
99
message %s
100
out_keys message
101
</match>
102
103
# Clear tag
104
<match clear>
105
@type null
106
</match>
Copied!

FAQ

With rewrite-tag-filter, logs are not forwarded. Why?

If you have following configuration, it doesn't work:
1
<match app.**>
2
@type rewrite_tag_filter
3
<rule>
4
key level
5
pattern (.+)
6
tag app.$1
7
</rule>
8
<match>
9
10
<match app.**>
11
@type forward
12
# ...
13
</match>
Copied!
In this case, rewrite_tag_filter causes infinite loop because fluentd's routing is executed from top to bottom. So you need to change tag like below:
1
<match app.**>
2
@type rewrite_tag_filter
3
<rule>
4
key level
5
pattern (.+)
6
tag level.app.$1
7
</rule>
8
<match>
9
10
<match level.app.**>
11
@type forward
12
# ...
13
</match>
Copied!
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 1yr ago