Comment on page
Parser Plugins
Fluentd has nine (9) types of plugins:
This article gives an overview of the Parser Plugin.
Sometimes, the
<parse>
directive for input plugins (e.g. in_tail
, in_syslog
, in_tcp
and in_udp
) cannot parse the user's custom data format (for example, a context-dependent grammar that can't be parsed with a regular expression). To address such cases, Fluentd has a pluggable system that enables the user to create their own parser formats.- for more information.
- From any input plugin that supports the
<parse>
directive, call the customplugin by its name.
Here is an example to read Nginx access logs using
in_tail
and parser_nginx
:<source>
@type tail
path /path/to/input/file
<parse>
@type nginx
keep_time_key true
</parse>
</source>
Note: When
td-agent
is launched by systemd, the default user of the td-agent
process is the td-agent
user. You must ensure that this user has read permission to the tailed /path/to/file
. For instance, on Ubuntu, the default Nginx access file /var/log/nginx/access.log
is mode 0640
and owned by www-data:adm
. In this case, several options are available to allow read access:- 1.Add the
td-agent
user to theadm
group, e.g. throughusermod -aG
, or - 2.to allow the invoking user to read the file without otherwise changing its permission bits or ownership.
- If you are familiar with
grok
patterns,grok-parser
plugin is useful. Use> 1.0.0
versions forfluentd
v0.14/v1.0. - If you need to parse multiple formats in one data stream,
multi-format-parser
is useful. - For protocol buffers.
- For Apache Avro.
Following plugins support
<parse>
directive:If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 2yr ago