Fluentd
Search…
1.0
multiline
The
multiline parser plugin parses multiline logs. This plugin is the multiline version of regexp parser.The
multiline parser parses log with formatN and format_firstline parameters. format_firstline is for detecting the start line of the multiline log. formatN, where N's range is [1..20], is the list of Regexp format for multiline log.Unlike other parser plugins, this plugin needs special code in input plugin e.g. handle
format_firstline. So, currently, in_tail plugin works with multiline but other input plugins do not work with it.Parameters
format_firstline
format_firstlineSpecifies the regexp pattern for the start line of multiple lines. Input plugin can skip the logs until
format_firstline is matched. Default is nil.If
format_firstline is not specified, the input plugin should store the unmatched new lines in the temporary buffer and try to match the buffered logs with each new line.formatN
formatNtype
default
version
string
nil0.14.0
It is a required parameter.
Specifies the regexp patterns. For readability, you can separate the regexp patterns into multiple
formatN parameters. See the Rails Log's example below. These patterns are joined and then construct a regexp pattern with multiline mode.Example
Rails Log
With this configuration:
1
<parse>
2
@type multiline
3
format_firstline /^Started/
4
format1 /Started (?<method>[^ ]+) "(?<path>[^"]+)" for (?<host>[^ ]+) at (?<time>[^ ]+ [^ ]+ [^ ]+)\n/
5
format2 /Processing by (?<controller>[^\u0023]+)\u0023(?<controller_method>[^ ]+) as (?<format>[^ ]+?)\n/
6
format3 /( Parameters: (?<parameters>[^ ]+)\n)?/
7
format4 / Rendered (?<template>[^ ]+) within (?<layout>.+) \([\d\.]+ms\)\n/
8
format5 /Completed (?<code>[^ ]+) [^ ]+ in (?<runtime>[\d\.]+)ms \(Views: (?<view_runtime>[\d\.]+)ms \| ActiveRecord: (?<ar_runtime>[\d\.]+)ms\)/
9
</parse>
Copied!
This incoming event:
1
Started GET "/users/123/" for 127.0.0.1 at 2013-06-14 12:00:11 +0900
2
Processing by UsersController#show as HTML
3
Parameters: {"user_id"=>"123"}
4
Rendered users/show.html.erb within layouts/application (0.3ms)
5
Completed 200 OK in 4ms (Views: 3.2ms | ActiveRecord: 0.0ms)
Copied!
is parsed as:
1
time:
2
1371178811 (2013-06-14 12:00:11 +0900)
3
​
4
record:
5
{
6
"method" :"GET",
7
"path" :"/users/123/",
8
"host" :"127.0.0.1",
9
"controller" :"UsersController",
10
"controller_method":"show",
11
"format" :"HTML",
12
"parameters" :"{ \"user_id\":\"123\"}",
13
...
14
}
Copied!
Java Stacktrace Log
With this configuration:
1
<parse>
2
@type multiline
3
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
4
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
5
</parse>
Copied!
These incoming events:
1
2013-3-03 14:27:33 [main] INFO Main - Start
2
2013-3-03 14:27:33 [main] ERROR Main - Exception
3
javax.management.RuntimeErrorException: null
4
at Main.main(Main.java:16) ~[bin/:na]
5
2013-3-03 14:27:33 [main] INFO Main - End
Copied!
are parsed as:
1
time:
2
2013-03-03 14:27:33 +0900
3
record:
4
{
5
"thread" :"main",
6
"level" :"INFO",
7
"message":" Main - Start"
8
}
9
​
10
time:
11
2013-03-03 14:27:33 +0900
12
record:
13
{
14
"thread" :"main",
15
"level" :"ERROR",
16
"message":" Main - Exception\njavax.management.RuntimeErrorException: null\n at Main.main(Main.java:16) ~[bin/:na]"
17
}
18
​
19
time:
20
2013-03-03 14:27:33 +0900
21
record:
22
{
23
"thread" :"main",
24
"level" :"INFO",
25
"message":" Main - End"
26
}
Copied!
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 11mo ago
Copy link