The goal of this article is to provide a concise post-installation guide to new Fluentd users. It is assumed that you've installed Fluentd through td-agent package.
A clean installation leaves you a td-agent instance running on a sample configuration file. You can edit the configuration file located at:
After editing this file, you need to restart td-agent using
$ sudo systemctl restart td-agent
By default, td-agent writes its operation logs to the following file:
If you want to make td-agent more verbose, read the article "Trouble Shooting".
In Fluentd, the most important part of data input/output is managed by plugins. Each plugin knows how to interface with a external endpoint and is responsible for managing a pipeline to convey data streams.
Plugins are named with a certain convention. For example, if it receives data and interfacing with Aapche Kafka, it's called
in_kafka. In the same way, if it publishes data and connects to MongoDB, it's called
The following snippet is an example configuration, which uses
in_forward plugin as an input source and
out_file plugin as an output endpoint.
<source>@type forwardport 9999</source><match app.**>@type filepath /var/log/app/data.logcompress gzip</match>
Fluentd manages plugins as Ruby gems, but stores these gems in a separate directory from where normal Ruby gems reside.
This is why you need to use a special program
td-agent-gem to manage Fluentd plugins. For example, the following command allows you to install the plugin to connect S3 (which contains both
$ sudo /usr/sbin/td-agent-gem install fluent-plugin-s3
See List Of All Plugins to explore available third-party plugins.
Note that a number of plugins are already included in the standard distribution of td-agent, so you may not need to install them manually.
A configuration file consists of a number of setting blocks (like
<source>). Each block contains a set of options for a specific data endpoint.
For example, if you want to create an endpoint to receive data from syslog, you need to add a
<source> block and set up its settings as follows.
<source>@type syslogport 5140tag system</source>
@type determines which plugin to use. You do not need prepend type prefix in this option (so
@type syslog, not
To add an output endpoint for data stream, you need to define a
<match> block. Syntactically,
<match> is slightly different from
<source> in the sense that it requires a filter expression as an argument.
For example, If you want to output events tagged with
debug.log, you need to write as below:
<match debug.log>@type kafka2port 5140brokers kafka-server:9092tag system# other parameters...</match>
You can use a wildcard character
* in the filter expression. For example,
If you want to catch all descendent tags, use double asterisks
**. For example,
debug.** matches not only
debug.log, but also
Read Configuration File Syntax for the full configuration syntax.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.