The parser
filter plugin "parses" string field in event records and mutates its event record with the parsed result.
It is included in the Fluentd's core.
<filter foo.bar>@type parserkey_name log<parse>@type regexpexpression /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)$/time_format %d/%b/%Y:%H:%M:%S %z</parse></filter>
filter_parser
uses built-in parser plugins and your own customized parser plugin, so you can reuse the predefined formats like apache2
, json
, etc. See Parser Plugin Overview for more details
With this example, if you receive this event:
time:injected time (depends on your input)record:{"log":"192.168.0.1 - - [05/Feb/2018:12:00:00 +0900] \"GET / HTTP/1.1\" 200 777"}
The parsed result will be:
time05/Feb/2018:12:00:00 +0900record:{"host":"192.168.0.1","user":"-","method":"GET","path":"/","code":"200","size":"777"}
​parser
​
​record_accessor
​
​compat_parameters
​
See Common Parameters.
This is a required subsection. Specifies the parser type and related parameter.
For more details, see Parse Section Configurations.
type | default | version |
string | required parameter | 0.14.9 |
Specifies the field name in the record to parse.
This parameter supports nested field access via record_accessor
syntax.
type | default | version |
bool | false | 0.14.9 |
Keeps the original event time in the parsed result.
type | default | version |
bool | false | 0.14.9 |
Keeps the original key-value pair in the parsed result.
<filter foo.bar>@type parserkey_name logreserve_data true<parse>@type json</parse></filter>
With above configuration, here is the result:
# input data: {"key":"value","log":"{\"user\":1,\"num\":2}"}# output data: {"key":"value","log":"{\"user\":1,\"num\":2}","user":1,"num":2}
Without reserve_data
, the result is:
# input data: {"key":"value","log":"{\"user\":1,\"num\":2}"}# output data: {"user":1,"num":2}
type | default | version |
bool | false | 1.2.2 |
Removes key_name
field when parsing is succeeded.
<filter foo.bar>@type parserkey_name logreserve_data trueremove_key_name_field true<parse>@type json</parse></filter>
With above configuration, here is the result:
# input data: {"key":"value","log":"{\"user\":1,\"num\":2}"}# output data: {"key":"value","user":1,"num":2}
type | default | version |
bool |
| 0.14.9 |
If true
, invalid string is replaced with safe characters and re-parse it.
type | default | version |
string | false | 0.14.9 |
Stores the parsed values with the specified key name prefix.
<filter foo.bar>@type parserkey_name logreserve_data trueinject_key_prefix data.<parse>@type json</parse></filter>
With above configuration, here is the result:
# input data: {"log": "{\"user\":1,\"num\":2}"}# output data: {"log":"{\"user\":1,\"num\":2}","data.user":1, "data.num":2}
type | default | version |
string | false | 0.14.9 |
Stores the parsed values as a hash value in a field.
<filter foo.bar>@type parserkey_name loghash_value_field parsed<parse>@type json</parse></filter>
With above configuration, result is below:
# input data: {"log": "{\"user\":1,\"num\":2}"}# output data: {"parsed":{"user":1,"num":2}}
type | default | version |
bool | true | 0.14.0 |
Emits invalid record to @ERROR
label. Invalid cases are:
key does not exist
the format is not matched
an unexpected error
You can rescue unexpected format logs in the @ERROR
label.
If you want to ignore these errors, set false
.
Since v1, parser
filter does not support suppress_parse_error_log
parameter because parser
filter uses the @ERROR
feature instead of internal logging to rescue invalid records. If you want to simply ignore invalid records, set emit_invalid_record_to_error false
.
See also emit_invalid_record_to_error
parameter.
​Filter Plugin Overview​
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.