Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Example Configurations
  • Plugin Helpers
  • Parameters
  • <parse> Section
  • key_name
  • reserve_time
  • reserve_data
  • remove_key_name_field
  • replace_invalid_sequence
  • inject_key_prefix
  • hash_value_field
  • emit_invalid_record_to_error
  • FAQ
  • suppress_parse_error_log is missing. What are the alternatives?
  • Learn More

Was this helpful?

  1. Filter Plugins

parser

PreviousgrepNextgeoip

Last updated 3 years ago

Was this helpful?

The parser filter plugin "parses" string field in event records and mutates its event record with the parsed result.

It is included in the Fluentd's core.

Example Configurations

<filter foo.bar>
  @type parser
  key_name log
  <parse>
    @type regexp
    expression /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)$/
    time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
</filter>

filter_parser uses built-in parser plugins and your own customized parser plugin, so you can reuse the predefined formats like apache2, json, etc. See for more details

With this example, if you receive this event:

time:
injected time (depends on your input)
record:
{"log":"192.168.0.1 - - [05/Feb/2018:12:00:00 +0900] \"GET / HTTP/1.1\" 200 777"}

The parsed result will be:

time
05/Feb/2018:12:00:00 +0900
record:
{"host":"192.168.0.1","user":"-","method":"GET","path":"/","code":"200","size":"777"}

Plugin Helpers

Parameters

<parse> Section

This is a required subsection. Specifies the parser type and related parameter.

key_name

type

default

version

string

required parameter

0.14.9

Specifies the field name in the record to parse.

reserve_time

type

default

version

bool

false

0.14.9

Keeps the original event time in the parsed result.

<filter foo.bar>
  @type parser
  key_name log
  reserve_time true
  <parse>
    @type json
  </parse>
</filter>

With above configuration, here is the result:

# input data:  {"key":"value","log":"{\"time\":1622473200,\"user\":1}"}

Above incoming event is parsed as:

time:
2021-06-01 00:00:00.000000000 +0900

record:
{
  "user": 1
}

The value of time field (1622473200) is reserved as event time (2021-06-01 00:00:00.000000000 +0900).

Without reserve_time, the result is:

# input data:  {"key":"value","log":"{\"time\":1622473200,\"user\":1}"}

Above incoming event is parsed as:

time:
2021-06-24 14:33:35.475115751 +0900 (It vary on parsed timestamp)

record:
{
  "user": 1
}

The value of parsed timestamp is set as event time. The value of time field is discarded.

reserve_data

type

default

version

bool

false

0.14.9

Keeps the original key-value pair in the parsed result.

<filter foo.bar>
  @type parser
  key_name log
  reserve_data true
  <parse>
    @type json
  </parse>
</filter>

With above configuration, here is the result:

# input data:  {"key":"value","log":"{\"user\":1,\"num\":2}"}
# output data: {"key":"value","log":"{\"user\":1,\"num\":2}","user":1,"num":2}

Without reserve_data, the result is:

# input data:  {"key":"value","log":"{\"user\":1,\"num\":2}"}
# output data: {"user":1,"num":2}

remove_key_name_field

type

default

version

bool

false

1.2.2

Removes key_name field when parsing is succeeded.

<filter foo.bar>
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

With above configuration, here is the result:

# input data:  {"key":"value","log":"{\"user\":1,\"num\":2}"}
# output data: {"key":"value","user":1,"num":2}

replace_invalid_sequence

type

default

version

bool

false

0.14.9

If true, invalid string is replaced with safe characters and re-parse it.

inject_key_prefix

type

default

version

string

false

0.14.9

Stores the parsed values with the specified key name prefix.

<filter foo.bar>
  @type parser
  key_name log
  reserve_data true
  inject_key_prefix data.
  <parse>
    @type json
  </parse>
</filter>

With above configuration, here is the result:

# input data:  {"log": "{\"user\":1,\"num\":2}"}
# output data: {"log":"{\"user\":1,\"num\":2}","data.user":1, "data.num":2}

hash_value_field

type

default

version

string

false

0.14.9

Stores the parsed values as a hash value in a field.

<filter foo.bar>
  @type parser
  key_name log
  hash_value_field parsed
  <parse>
    @type json
  </parse>
</filter>

With above configuration, result is below:

# input data:  {"log": "{\"user\":1,\"num\":2}"}
# output data: {"parsed":{"user":1,"num":2}}

emit_invalid_record_to_error

type

default

version

bool

true

0.14.0

Emits invalid record to @ERROR label. Invalid cases are:

  • key does not exist

  • the format is not matched

  • an unexpected error

You can rescue unexpected format logs in the @ERROR label.

If you want to ignore these errors, set false.

FAQ

suppress_parse_error_log is missing. What are the alternatives?

Since v1, parser filter does not support suppress_parse_error_log parameter because parser filter uses the @ERROR feature instead of internal logging to rescue invalid records. If you want to simply ignore invalid records, set emit_invalid_record_to_error false.

See also emit_invalid_record_to_error parameter.

Learn More

See .

For more details, see .

This parameter supports nested field access via .

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

Parser Plugin Overview
parser
record_accessor
compat_parameters
Common Parameters
Parse Section Configurations
Filter Plugin Overview
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
record_accessor syntax