Fluentd has 8 types of plugins: Input, Parser, Filter, Output, Formatter, Storage, Service Discovery and Buffer. This article gives an overview of Parser Plugin.

Overview

Sometimes, the <parse> directive for input plugins (ex: in_tail, in_syslog, in_tcp and in_udp) cannot parse the user's custom data format (for example, a context-dependent grammar that can't be parsed with a regular expression). To address such cases. Fluentd has a pluggable system that enables the user to create their own parser formats.

How To Use

• Write a custom format plugin. See here for more information.

• From any input plugin that supports the <parse> directive, call

  the custom plugin by its name.

Here is a simple example to read Nginx access logs using in_tail and parser_nginx:

<source>
  @type tail
  path /path/to/input/file
  <parse>
    @type nginx
    keep_time_key true
  </parse>
</source>

List of Built-in Parsers

• ​regexp​

• ​apache2​

• ​apache_error​

• ​nginx​

• ​syslog​

• ​csv​

• ​tsv​

• ​ltsv​

• ​json​

• ​msgpack​

• ​multiline​

• ​none​

3rd party Parsers

• ​grok​

If you are familiar with grok patterns, grok-parser plugin is useful. Use > 1.0.0 versions for fluentd v0.14/v1.0.

• ​multi-format-parser​

If you need to parse multiple formats in one data stream, multi-format-parser is useful.

• ​protobuf​

For protocol buffers.

List of Core Input Plugins with Parser support

with <parse> directive.

• ​in_tail​

• ​in_tcp​

• ​in_udp​

• ​in_syslog​

• ​in_http​

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.