Fluentd
Search…
multiline
The
multiline parser plugin parses multiline logs. This plugin is multiline version of regexp parser.The
multiline parser parses log with formatN and format_firstline parameters. format_firstline is for detecting start line of multiline log. formatN, N's range is 1..20, is the list of Regexp format for multiline log. Unlike other parser plugins, this plugin needs special code in input plugin, e.g. handle format_firstline. So currently, in_tail plugin works with `multiline` but other input plugins don't work with `multiline`.Parameters
time_key
Specify the field for event time. Default is
time.time_format
Specify time format for
time_key.format_firstline
Specify regexp pattern for start line of multiple lines. Input plugin can skip the logs until
format_firstline is matched. Default is nil.If
format_firstline is not specified, input plugin should store unmatched new lines in temporary buffer and try to match buffered logs with each new line.formatN
Specify regexp patterns. For readability, you can separate regexp patterns into multiple regexpN parameters, See "Rails log" example. These patterns are joined and constructs regexp pattern with multiline mode.
keep_time_key
If you want to keep time field in the record, set
true. Default is false.Example
Rails log
1
format multiline
2
format_firstline /^Started/
3
format1 /Started (?<method>[^ ]+) "(?<path>[^"]+)" for (?<host>[^ ]+) at (?<time>[^ ]+ [^ ]+ [^ ]+)\n/
4
format2 /Processing by (?<controller>[^\u0023]+)\u0023(?<controller_method>[^ ]+) as (?<format>[^ ]+?)\n/
5
format3 /( Parameters: (?<parameters>[^ ]+)\n)?/
6
format4 / Rendered (?<template>[^ ]+) within (?<layout>.+) \([\d\.]+ms\)\n/
7
format5 /Completed (?<code>[^ ]+) [^ ]+ in (?<runtime>[\d\.]+)ms \(Views: (?<view_runtime>[\d\.]+)ms \| ActiveRecord: (?<ar_runtime>[\d\.]+)ms\)/
Copied!
With this configuration:
1
Started GET "/users/123/" for 127.0.0.1 at 2013-06-14 12:00:11 +0900
2
Processing by UsersController#show as HTML
3
Parameters: {"user_id"=>"123"}
4
Rendered users/show.html.erb within layouts/application (0.3ms)
5
Completed 200 OK in 4ms (Views: 3.2ms | ActiveRecord: 0.0ms)
Copied!
This incoming event is parsed as:
1
time:
2
1371178811 (2013-06-14 12:00:11 +0900)
3
​
4
record:
5
{
6
"method" :"GET",
7
"path" :"/users/123/",
8
"host" :"127.0.0.1",
9
"controller" :"UsersController",
10
"controller_method":"show",
11
"format" :"HTML",
12
"parameters" :"{ \"user_id\":\"123\"}",
13
...
14
}
Copied!
Java stacktrace log
1
format multiline
2
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
3
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
Copied!
With this configuration:
1
2013-3-03 14:27:33 [main] INFO Main - Start
2
2013-3-03 14:27:33 [main] ERROR Main - Exception
3
javax.management.RuntimeErrorException: null
4
at Main.main(Main.java:16) ~[bin/:na]
5
2013-3-03 14:27:33 [main] INFO Main - End
Copied!
These incoming events are parsed as:
1
time:
2
2013-03-03 14:27:33 +0900
3
record:
4
{
5
"thread" :"main",
6
"level" :"INFO",
7
"message":" Main - Start"
8
}
9
​
10
time:
11
2013-03-03 14:27:33 +0900
12
record:
13
{
14
"thread" :"main",
15
"level" :"ERROR",
16
"message":" Main - Exception\njavax.management.RuntimeErrorException: null\n at Main.main(Main.java:16) ~[bin/:na]"
17
}
18
​
19
time:
20
2013-03-03 14:27:33 +0900
21
record:
22
{
23
"thread" :"main",
24
"level" :"INFO",
25
"message":" Main - End"
26
}
Copied!
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 1yr ago