parser
The filter_parser
filter plugin "parses" string field in event records and mutates its event record with parsed result.
Example Configurations
filter_parser
is included in Fluentd's core since v0.12.29. No installation required. If you want to use filter_parser
with lower fluentd versions, need to install fluent-plugin-parser
.
filter_parser
has just same with in_tail
about format
and time_format
:
<filter foo.bar>
@type parser
format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)$/
time_format %d/%b/%Y:%H:%M:%S %z
key_name message
</filter>
filter_parser
uses built-in parser plugins and your own customized parser plugin, so you can re-use pre-defined format like apache
, json
and etc. See document page for more details: Parser Plugin Overview
Parameters
format
This is required parameter. Specify parser format or regexp pattern.
key_name
This is required parameter. Specify field name in the record to parse.
reserve_data
Keep original key-value pair in parsed result. Default is false
.
<filter foo.bar>
@type parser
format json
key_name log
reserve_data true
</filter>
With above configuration, result is below:
# input data: {"key":"value","log":"{\"user\":1,\"num\":2}"}
# output data: {"key":"value","log":"{\"user\":1,\"num\":2}","user":1,"num":2}
Without reserve_data
, result is below
# input data: {"key":"value","log":"{\"user\":1,\"num\":2}"}
# output data: {"user":1,"num":2}
suppress_parse_error_log
If true
, a plugin suppresses pattern not match
warning log. Default is false
.
This parameter is useful for parsing mixed logs and you want to ignore non target lines.
ignore_key_not_exist
Ignore "key not exist" log. Default is false
.
Useful case is same with suppress_parse_error_log
.
replace_invalid_sequence
If true
, invalid string is replaced with safe characters and re-parse it. Default is false
.
inject_key_prefix
Store parsed values with specified key name prefix. Default is nil
.
<filter foo.bar>
@type parser
format json
key_name log
reserve_data true
inject_key_prefix data.
</filter>
With above configuration, result is below:
# input data: {"log": "{\"user\":1,\"num\":2}"}
# output data: {"log":"{\"user\":1,\"num\":2}","data.user":1, "data.num":2}
hash_value_field
Store parsed values as a hash value in a field. Default is nil
.
<filter foo.bar>
@type parser
format json
key_name log
hash_value_field parsed
</filter>
With above configuration, result is below:
# input data: {"log": "{\"user\":1,\"num\":2}"}
# output data: {"parsed":{"user":1,"num":2}}
time_parse
If false, time parsing is disabled in the parser. Default is true.
emit_invalid_record_to_error
Emit invalid record to @ERROR
label. Default is false
. Invalid cases are
key not exist
format is not matched
unexpected error
You can rescue unexpected format logs in @ERROR
label.
Learn More
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last updated
Was this helpful?