Some of the Fluentd plugins support the <parse>
section to specify how to parse the raw data.
The parse section can be under <source>
, <match>
or <filter>
section. It is enabled for the plugins that support parser plugin features.
<source>@type tail# ...<parse># ...</parse></source>
The @type
parameter of <parse>
section specifies the type of the parser plugin. Fluentd core bundles some useful parser plugins.
<parse>@type apache2</parse>
Third-party plugins may also be installed and configured.
For more details, see plugins documentation.
The @type
parameter specifies the type of the parser plugin.
<parse>@type regexp# ...</parse>
Here's the list of built-in parser plugins:
​regexp
​
​apache2
​
​apache_error
​
​nginx
​
​syslog
​
​csv
​
​tsv
​
​ltsv
​
​json
​
​multiline
​
​none
​
The default value of the following parameters will be overridden by the individual parser plugins:
types
(hash) (optional): Specify types for converting field into another
type. See below "The detail of types parameter" section.
Default: nil
string-based hash: `field1:type, field2:type, field3:type:option,
field4:type:option`
JSON format: `{"field1":"type", "field2":"type", "field3":"type:option",
"field4":"type:option"}`
example: types user_id:integer,paid:bool,paid_usd_amount:float
null_value_pattern
(string) (optional): Specify null value pattern.
Default: nil
null_empty_string
(bool) (optional): If true
, empty string field is
replaced with nil
.
Default: false
estimate_current_event
(bool) (optional): If true
, use
Fluent::EventTime.now
(current time) as a timestamp when time_key
is
specified.
Default: true
keep_time_key
(bool) (optional): If true
, keep time field in the
record.
Default: false
timeout
(time) (optional): Specify timeout for parse
processing. This
is mainly for detecting wrong regexp pattern.
Default: nil
For the types
parameter, the following types are supported:
string
: Converts the field into String
type. This uses to_s
method for conversion.
bool
: Converts the string "true"
, "yes"
or "1"
into true
. Otherwise, false
.
integer
(not int
): Converts the field into the Integer
type. This uses to_i
method for conversion. For example, the string "1000"
converts into 1000
.
float
: Converts the field into Float
type. This uses to_f
method for conversion. For example, the string "7.45"
converts into 7.45
.
time
: Converts the field into Fluent::EventTime
type. This uses Fluentd time parser for conversion. For the time
type, the third field specifies the time format similar to time_format
.
date:time:%d/%b/%Y:%H:%M:%S %z # for string with time formatdate:time:unixtime # for integer timedate:time:float # for float time
See time_type
and time_format
parameters in Time parameters
section.
array
: Converts the string field into Array
type. For the array
type, the third field specifies the delimiter (the default is comma ","
). For example, if a field item_ids
contains the value "3,4,5"
, types item_ids:array
parses it as ["3", "4", "5"]
. Alternatively, if the value is "Adam|Alice|Bob"
, types item_ids:array:|
parses it as ["Adam", "Alice", "Bob"]
.
time_type
(enum) (optional): parses/formats value according to this
type
Default: float
Available values: float
, unixtime
, string
float
: seconds from Epoch + nano seconds (e.g.
1510544836.154709804)
unixtime
: seconds from Epoch (e.g. 1510544815)
string
: use format specified by time_format
, local time or time
zone
time_format
(string) (optional): processes value according to the
specified format. This is available only when time_type
is string
.
Default: nil
Available time format:
For more details about formatting, see
​Time#strftime
.
For more details about parsing, see
​Time.strptime
.
%iso8601
(only for parsing)
Use %N
to parse/format with sub-second precision, because
​strptime
does not support
%3N
, %6N
, %9N
, and %L
.
localtime
(bool) (optional): if true
, uses local time. Otherwise,
UTC is used. This is exclusive with utc
.
Default: true
utc
(bool) (optional): if true
, uses UTC. Otherwise, local time is
used. This is exclusive with localtime
.
Default: false
timezone
(string) (optional): uses the specified timezone. One can
parse/format the time value in the specified timezone format.
Default: nil
Available time zone format:
[+-]HH:MM
(e.g. "+09:00") (recommended)
[+-]HHMM
(e.g. "+0900")
[+-]HH
(e.g. "+09")
Region/Zone (e.g. Asia/Tokyo
)
Region/Zone/Zone (e.g. America/Argentina/Buenos_Aires
)
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.