Parser Plugins

Search…
Parser Plugins
Fluentd has nine (9) types of plugins:
​Input​
​Parser​
​Filter​
​Output​
​Formatter​
​Storage​
​Service Discovery​
​Buffer​
​Metrics​
This article gives an overview of the Parser Plugin.
Overview
Sometimes, the <parse> directive for input plugins (e.g. in_tail, in_syslog, in_tcp and in_udp) cannot parse the user's custom data format (for example, a context-dependent grammar that can't be parsed with a regular expression). To address such cases, Fluentd has a pluggable system that enables the user to create their own parser formats.
How To Use
Write a custom format plugin. See here​
for more information.
From any input plugin that supports the <parse> directive, call the custom
plugin by its name.
Here is an example to read Nginx access logs using in_tail and parser_nginx:
1
<source>
2
  @type tail
3
  path /path/to/input/file
4
  <parse>
5
    @type nginx
6
    keep_time_key true
7
  </parse>
8
</source>
Copied!
Note: When td-agent is launched by systemd, the default user of the td-agent process is the td-agent user. You must ensure that this user has read permission to the tailed /path/to/file. For instance, on Ubuntu, the default Nginx access file /var/log/nginx/access.log is mode 0640 and owned by www-data:adm. In this case, several options are available to allow read access:
1.Add the td-agent user to the adm group, e.g. through usermod -aG, or
2.Use the cap_dac_read_search capability​
to allow the invoking user to read the file without otherwise changing its permission bits or ownership.
List of Built-in Parsers
​regexp​
​apache2​
​apache_error​
​nginx​
​syslog​
​csv​
​tsv​
​ltsv​
​json​
​msgpack​
​multiline​
​none​
Third-party Parsers
​grok​
If you are familiar with grok patterns, grok-parser plugin is useful. Use > 1.0.0 versions for fluentd v0.14/v1.0.
​multi-format-parser​
If you need to parse multiple formats in one data stream, multi-format-parser is useful.
​protobuf​
For protocol buffers.
​avro​
For Apache Avro.
List of Core Input Plugins with Parser support
Following plugins support <parse> directive:
​in_tail​
​in_tcp​
​in_udp​
​in_syslog​
​in_http​
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
stdout
regexp
Last modified 1mo ago
Copy link
Contents
Overview
How To Use
List of Built-in Parsers
Third-party Parsers
List of Core Input Plugins with Parser support