Config: Parse Section
Some of the Fluentd plugins support the
<parse>
section to specify how to parse the raw data.The parse section can be under
<source>
, <match>
or <filter>
section. It is enabled for the plugins that support parser plugin features.<source>
@type tail
# ...
<parse>
# ...
</parse>
</source>
The
@type
parameter of <parse>
section specifies the type of the parser plugin. Fluentd core bundles some useful parser plugins.<parse>
@type apache2
</parse>
Third-party plugins may also be installed and configured.
For more details, see plugins documentation.
The
@type
parameter specifies the type of the parser plugin.<parse>
@type regexp
# ...
</parse>
Here's the list of built-in parser plugins:
The default value of the following parameters will be overridden by the individual parser plugins:
types
(hash) (optional): Specify types for converting field into anothertype. See below "The detail of types parameter" section.- Default:
nil
- string-based hash: `field1:type, field2:type, field3:type:option,field4:type:option`
- JSON format: `{"field1":"type", "field2":"type", "field3":"type:option","field4":"type:option"}`
- example:
types user_id:integer,paid:bool,paid_usd_amount:float
null_value_pattern
(regexp) (optional): Specify null value pattern.- Default:
nil
null_empty_string
(bool) (optional): Iftrue
, empty string field isreplaced withnil
.- Default:
false
estimate_current_event
(bool) (optional): Iftrue
, useFluent::EventTime.now
(current time) as a timestamp whentime_key
isspecified.- Default:
true
keep_time_key
(bool) (optional): Iftrue
, keep time field in therecord.- Default:
false
timeout
(time) (optional): Specify timeout forparse
processing. Thisis mainly for detecting wrong regexp pattern.- Default:
nil
For the
types
parameter, the following types are supported:string
: Converts the field intoString
type. This usesto_s
method for conversion.bool
: Converts the string"true"
,"yes"
or"1"
intotrue
. Otherwise,false
.integer
(notint
): Converts the field into theInteger
type. This usesto_i
method for conversion. For example, the string"1000"
converts into1000
.float
: Converts the field intoFloat
type. This usesto_f
method for conversion. For example, the string"7.45"
converts into7.45
.time
: Converts the field intoFluent::EventTime
type. This uses Fluentd time parser for conversion. For thetime
type, the third field specifies the time format similar totime_format
.date:time:%d/%b/%Y:%H:%M:%S %z # for string with time formatdate:time:unixtime # for integer timedate:time:float # for float timeSeetime_type
andtime_format
parameters inTime parameters
section.array
: Converts the string field intoArray
type. For thearray
type, the third field specifies the delimiter (the default is comma","
). For example, if a fielditem_ids
contains the value"3,4,5"
,types item_ids:array
parses it as["3", "4", "5"]
. Alternatively, if the value is"Adam|Alice|Bob"
,types item_ids:array:|
parses it as["Adam", "Alice", "Bob"]
.
time_type
(enum) (optional): parses/formats value according to thistype- Default:
string
- Available values:
float
,unixtime
,string
,mixed
float
: seconds from Epoch + nano seconds (e.g.1510544836.154709804)unixtime
: seconds from Epoch (e.g. 1510544815)string
: use format specified bytime_format
, local time or timezonemixed
: enabletime_format_fallbacks
option. (Since Fluentd v1.12.2)
time_format
(string) (optional): processes value according to thespecified format. This is available only whentime_type
isstring
.- Default:
nil
- Available time format:
- For more details about formatting, see
- For more details about parsing, see
%iso8601
(only for parsing)- Use
%N
to parse/format with sub-second precision, because%3N
,%6N
,%9N
, and%L
.
localtime
(bool) (optional): iftrue
, uses local time. Otherwise,UTC is used. This is exclusive withutc
.- Default:
true
utc
(bool) (optional): iftrue
, uses UTC. Otherwise, local time isused. This is exclusive withlocaltime
.- Default:
false
timezone
(string) (optional): uses the specified timezone. One canparse/format the time value in the specified timezone format.- Default:
nil
- Available time zone format:
- 1.
[+-]HH:MM
(e.g. "+09:00") (recommended) - 2.
[+-]HHMM
(e.g. "+0900") - 3.
[+-]HH
(e.g. "+09") - 4.Region/Zone (e.g.
Asia/Tokyo
) - 5.Region/Zone/Zone (e.g.
America/Argentina/Buenos_Aires
)
time_format_fallbacks
() (optional): uses the specified time format as a fallback in the specified order.You can parse undetermined time format by usingtime_format_fallbacks
. This options is enabled whentime_type
ismixed
.- Default:
nil
time_type mixedtime_format unixtimetime_format_fallbacks %iso8601In the above use case, the timestamp is parsed asunixtime
at first, if it fails, then it is parsed as%iso8601
secondary. Note thattime_format_fallbacks
is the last resort to parse mixed timestamp format. There is a performance penalty (Typically, N fallbacks are specified intime_format_fallbacks
and if the last specified format is used as a fallback, N times slower in the worst case).
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 1yr ago