Parser Plugins

Fluentd has nine (9) types of plugins:

• Input

• Parser

• Filter

• Output

• Formatter

• Storage

• Service Discovery

• Buffer

• Metrics

This article gives an overview of the Parser Plugin.

Overview

Sometimes, the <parse> directive for input plugins (e.g. in_tail, in_syslog, in_tcp and in_udp) cannot parse the user's custom data format (for example, a context-dependent grammar that can't be parsed with a regular expression). To address such cases, Fluentd has a pluggable system that enables the user to create their own parser formats.

How To Use

• Write a custom format plugin. See here

  for more information.

• From any input plugin that supports the <parse> directive, call the custom

  plugin by its name.

Here is an example to read Nginx access logs using in_tail and parser_nginx:

<source>
  @type tail
  path /path/to/input/file
  <parse>
    @type nginx
    keep_time_key true
  </parse>
</source>

Note: When td-agent is launched by systemd, the default user of the td-agent process is the td-agent user. You must ensure that this user has read permission to the tailed /path/to/file. For instance, on Ubuntu, the default Nginx access file /var/log/nginx/access.log is mode 0640 and owned by www-data:adm. In this case, several options are available to allow read access:

1. Add the td-agent user to the adm group, e.g. through usermod -aG, or

2. Use the cap_dac_read_search capability

   to allow the invoking user to read the file without otherwise changing its permission bits or ownership.

List of Built-in Parsers

• regexp
• apache2
• apache_error
• nginx
• syslog
• csv
• tsv
• ltsv
• json
• msgpack
• multiline
• none

Third-party Parsers

• grok

  If you are familiar with grok patterns, grok-parser plugin is useful. Use > 1.0.0 versions for fluentd v0.14/v1.0.

• multi-format-parser

  If you need to parse multiple formats in one data stream, multi-format-parser is useful.

• protobuf

  For protocol buffers.

• avro

  For Apache Avro.

List of Core Input Plugins with Parser support

Following plugins support <parse> directive:

• in_tail
• in_tcp
• in_udp
• in_syslog
• in_http

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.