Config File Syntax
This article describes the basic concepts of Fluentd's configuration file syntax.
Here is a brief overview of the life of a Fluentd event to help you understand the rest of this page:
The configuration file allows the user to control the input and output behavior of Fluentd by (1) selecting input and output plugins and (2) specifying the plugin parameters. The file is required for Fluentd to operate properly.
If you installed Fluentd using the td-agent packages, the config file is located at /etc/td-agent/td-agent.conf.
$ sudo vi /etc/td-agent/td-agent.conf
If you installed Fluentd using the Ruby Gem, you can create the configuration file using the following commands. Sending a SIGHUP signal will reload the config file.
$ sudo fluentd --setup /etc/fluent
$ sudo vi /etc/fluent/fluent.conf
You can change default configuration file location via
FLUENT_CONF
. For example, /etc/td-agent/td-agent.conf
is specified via FLUENT_CONF
inside td-agent scripts.Fluentd assumes configuration file is UTF-8 or ASCII.
The configuration file consists of the following directives:
- 1.source directives determine the input sources.
- 2.match directives determine the output destinations.
- 3.filter directives determine the event processing pipelines.
- 4.system directives set system wide configuration.
- 5.label directives group the output and filter for internalrouting
- 6.include directives include other files.
Let's actually create a configuration file step by step.
Fluentd's input sources are enabled by selecting and configuring the desired input plugins using source directives. Fluentd's standard input plugins include
http
and forward
. http
turns fluentd into an HTTP endpoint to accept incoming HTTP messages whereas forward
turns fluentd into a TCP endpoint to accept TCP packets. Of course, it can be both at the same time (You can add as many sources as you wish)# Receive events from 24224/tcp
# This is used by log forwarding and the fluent-cat command
<source>
@type forward
port 24224
</source>
# http://this.host:9880/myapp.access?json={"event":"data"}
<source>
@type http
port 9880
</source>
Each source directive must include a
@type
parameter. The @type
parameter specifies which input plugin to use.The
source
submits events into the Fluentd's routing engine. An event consists of three entities: tag, time and record. The tag is a string separated by '.'s (e.g. myapp.access), and is used as the directions for Fluentd's internal routing engine. The time field is specified by input plugins, and it must be in the Unix time format. The record is a JSON object. Fluentd accepts all non-period characters as a part of a tag. However, since the tag is sometimes used in a different context by output destinations (e.g., table name, database name, key name, etc.), it is strongly recommended that you stick to the lower-case alphabets, digits and underscore, e.g., ^[a-z0-9_]+$
.In the example above, the HTTP input plugin submits the following event:
# generated by http://this.host:9880/myapp.access?json={"event":"data"}
tag: myapp.access
time: (current time)
record: {"event":"data"}
You can add new input sources by writing your own plugins. For further information regarding Fluentd's input sources, please refer to the Input Plugin Overview article.
The "match" directive looks for events with matching tags and processes them. The most common use of the match directive is to output events to other systems (for this reason, the plugins that correspond to the match directive are called "output plugins"). Fluentd's standard output plugins include
file
and forward
. Let's add those to our configuration file.# Receive events from 24224/tcp
# This is used by log forwarding and the fluent-cat command
<source>
@type forward
port 24224
</source>
# http://this.host:9880/myapp.access?json={"event":"data"}
<source>
@type http
port 9880
</source>
# Match events tagged with "myapp.access" and
# store them to /var/log/fluent/access.%Y-%m-%d
# Of course, you can control how you partition your data
# with the time_slice_format option.
<match myapp.access>
@type file
path /var/log/fluent/access
</match>
Each match directive must include a match pattern and a
@type
parameter. Only events with a tag matching the pattern will be sent to the output destination (in the above example, only the events with the tag "myapp.access" is matched. See the section below for more advanced usage). The @type
parameter specifies the output plugin to use.Just like input sources, you can add new output destinations by writing your own plugins. For further information regarding Fluentd's output destinations, please refer to the Output Plugin Overview article.
The "filter" directive has same syntax as "match" but "filter" could be chained for processing pipeline. Using filters, event flow is like below:
Input -> filter 1 -> ... -> filter N -> Output
Let's add standard
record_transformer
filter to "match" example.# http://this.host:9880/myapp.access?json={"event":"data"}
<source>
@type http
port 9880
</source>
<filter myapp.access>
@type record_transformer
<record>
host_param "#{Socket.gethostname}"
</record>
</filter>
<match myapp.access>
@type file
path /var/log/fluent/access
</match>
Received event,
{"event":"data"}
, goes to record_transformer
filter first. record_transformer
adds "host_param" field to event and filtered event, {"event":"data","host_param":"webserver1"}
, goes to file
output.You can also add new filters by writing your own plugins. For further information regarding Fluentd's filter destinations, please refer to the Filter Plugin Overview article.
Following configurations are set by system directive. You can set same configurations by fluentd options:
- log_level
- suppress_repeated_stacktrace
- emit_error_log_interval
- suppress_config_dump
- without_source
- process_name (only available in system directive. No fluentdoption)
Here is an example:
<system>
# equal to -qq option
log_level error
# equal to --without-source option
without_source
# ...
</system>
If set this parameter, fluentd's supervisor and worker process names are changed.
<system>
process_name fluentd1
</system>
If we have this configuration,
ps
command shows the following result:% ps aux | grep fluentd1
foo 45673 0.4 0.2 2523252 38620 s001 S+ 7:04AM 0:00.44 worker:fluentd1
foo 45647 0.0 0.1 2481260 23700 s001 S+ 7:04AM 0:00.40 supervisor:fluentd1
This feature requires ruby 2.1 or later.
The "label" directive groups filter and output for internal routing. "label" reduces the complexity of tag handling.
Here is a configuration example. "label" is built-in plugin parameter so
@
prefix is needed.<source>
@type forward
</source>
<source>
@type tail
@label @SYSTEM
</source>
<filter access.**>
@type record_transformer
<record>
# ...
</record>
</filter>
<match **>
@type elasticsearch
# ...
</match>
<label @SYSTEM>
<filter var.log.middleware.**>
@type grep
# ...
</filter>
<match **>
@type s3
# ...
</match>
</label>
In this configuration,
forward
events are routed to record_transformer
filter / elasticsearch
output and in_tail
events are routed to grep
filter / s3
output inside @SYSTEM
label."label" is useful for event flow separation without tag prefix.
@ERROR
label is a built-in label used for error record emitted by plugin's emit_error_event
API.If you set
<label @ERROR>
in the configuration, events are routed to this label when emit related error, e.g. buffer is full or invalid record.(6) Re-use your config: the "## include" directive
Directives in separate configuration files can be imported using the include directive:
# Include config files in the ./config.d directory
@include config.d/*.conf
The include directive supports regular file path, glob pattern, and http URL conventions:
# absolute path
@include /path/to/config.conf
# if using a relative path, the directive will use
# the dirname of this config file to expand the path
@include extra.conf
# glob match pattern
@include config.d/*.conf
# http
@include http://example.com/fluent.conf
Note for glob pattern, files are expanded in the alphabetical order. If you have
a.conf
and b.conf
, fluentd parses a.conf
first. But you should not write the configuration depends on this order. It is so error prone. Please separate @include
for safety.# If you have a.conf,b.conf,...,z.conf and a.conf / z.conf are important...
# This is bad
@include *.conf
# This is good
@include a.conf
@include config.d/*.conf
@include z.conf
As described above, Fluentd allows you to route events based on their tags. Although you can just specify the exact tag to be matched (like
<filter app.log>
), there are a number of techniques you can use to manage the data flow more efficiently.The following match patterns can be used in
<match>
and <filter>
tags.*
matches a single tag part.- For example, the pattern
a.*
matchesa.b
, but does not matcha
ora.b.c
**
matches zero or more tag parts.- For example, the pattern
a.**
matchesa
,a.b
anda.b.c
{X,Y,Z}
matches X, Y, or Z, where X, Y, and Z are match patterns.- For example, the pattern
{a,b}
matchesa
andb
, but does not matchc
- This can be used in combination with the
*
or**
patterns. Examples includea.{b,c}.*
anda.{b,c.**}
- When multiple patterns are listed inside a single tag (delimited by one or more whitespaces), it matches any of the listed patterns. For example:
- The patterns
<match a b>
matcha
andb
. - The patterns
<match a.** b.*>
matcha
,a.b
,a.b.c
(from the first pattern) andb.d
(from the second pattern).
Fluentd tries to match tags in the order that they appear in the config file. So if you have the following configuration:
# ** matches all tags. Bad :(
<match **>
@type blackhole_plugin
</match>
<match myapp.access>
@type file
path /var/log/fluent/access
</match>
then
myapp.access
is never matched. Wider match patterns should be defined after tight match patterns.<match myapp.access>
@type file
path /var/log/fluent/access
</match>
# Capture all unmatched tags. Good :)
<match **>
@type blackhole_plugin
</match>
Of course, if you use two same patterns, second
match
is never matched. If you want to send events to multiple outputs, consider out_copy plugin.The common pitfall is when you put a
<filter>
block after <match>
. It will never work as supposed, since events never go through the filter for the reason explained above.# You should NOT put this <filter> block after the <match> block below.
# If you do, Fluentd will just emit events without applying the filter.
<filter myapp.access>
@type record_transformer
...
</filter>
<match myapp.access>
@type file
path /var/log/fluent/access
</match>
Each Fluentd plugin has a set of parameters. For example, in_tail has parameters such as
rotate_wait
and pos_file
. Each parameter has a specific type associated with it. They are defined as follows:Each parameter's type should be documented. If not, please let the plugin author know.
string
type: the field is parsed as a string. This is the most"generic" type, where each plugin decides how to process the string.string
has 3 literals, non-quoted one line string,'
quotedstring and"
quoted string.
integer
type: the field is parsed as an integer.float
type: the field is parsed as a float.size
type: the field is parsed as the number of bytes. There areseveral notational variations:- If the value matches
<INTEGER>k
or<INTEGER>K
, then thevalue is the INTEGER number of kilobytes. - If the value matches
<INTEGER>m
or<INTEGER>M
, then thevalue is the INTEGER number of megabytes. - If the value matches
<INTEGER>g
or<INTEGER>G
, then thevalue is the INTEGER number of gigabytes. - If the value matches
<INTEGER>t
or<INTEGER>T
, then thevalue is the INTEGER number of terabytes. - Otherwise, the field is parsed as integer, and that integer isthe number of bytes.
time
type: the field is parsed as a time duration.- If the value matches
<INTEGER>s
, then the value is the INTEGERseconds. - If the value matches
<INTEGER>m
, then the value is the INTEGERminutes. - If the value matches
<INTEGER>h
, then the value is the INTEGERhours. - If the value matches
<INTEGER>d
, then the value is the INTEGERdays. - Otherwise, the field is parsed as float, and that float is thenumber of seconds. This option is useful for specifyingsub-second time durations such as "0.1" (=0.1 second = 100ms).
array
type: the field is parsed as a JSON array. It also supportsshorthand syntax. These are same values.- normal:
["key1", "key2"]
- shorthand:
key1,key2
hash
type: the field is parsed as a JSON object. It also supportsshorthand syntax. These are same values.- normal:
{"key1":"value1", "key2":"value2"}
- shorthand:
key1:value1,key2:value2
array
and hash
are JSON because almost all programming languages and infrastructure tools can generate JSON value easily than unusual format.These parameters are system reserved and it has
@
prefix.@type
: Specify plugin type@id
: Specify plugin id. in_monitor_agent uses this value forplugin_id field
type
, id
and log_level
are supported for backward compatibility.You can check your configuration without plugins start by specifying
--dry-run
option.$ fluentd --dry-run -c fluent.conf
This section describes useful features in configuration format.
You can write multi line value for
"
quoted string, array and hash values.str_param "foo # This line is converted to "foo\nbar". NL is kept in the parameter
bar"
array_param [
"a", "b"
]
hash_param {
"k":"v",
"k1":10
}
Fluentd assumes
[
or {
is a start of array / hash. So if you want to set [
or {
started but non-json parameter, please use '
or "
.Example1: mail plugin:
<match **>
@type mail
subject "[CRITICAL] foo's alert system"
</match>
Example2: map plugin:
<match tag>
@type map
map '[["code." + tag, time, { "code" => record["code"].to_i}], ["time." + tag, time, { "time" => record["time"].to_i}]]'
multi true
</match>
We will remove this restriction with configuration parser improvement.
You can evaluate the Ruby code with
#{}
in "
quoted string. This is useful for setting machine information like hostname.host_param "#{Socket.gethostname}" # host_param is actual hostname like `webserver1`.
env_param "foo-#{ENV["FOO_BAR"]}" # NOTE that foo-"#{ENV["FOO_BAR"]}" doesn't work.
config-xxx mixins use "${}", not "#{}". These embedded configurations are two different things.
\
is interpreted as escape character. You need \
for setting "
, \r
, \n
, \t
, \
or several characters in double-quoted string literal.str_param "foo\nbar" # \n is interpreted as actual LF character
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 3yr ago