Config File Syntax (YAML)
This article describes the basic concepts of Fluentd configuration file syntax for yaml format.
Here is a brief overview of the lifecycle of a Fluentd event to help you understand the rest of this page:
The configuration file allows the user to control the input and output behavior of Fluentd by 1) selecting input and output plugins; and, 2) specifying the plugin parameters. The file is required for Fluentd to operate properly.
Fluentd starts to support YAML configuration format but this is not 1-by-1 correspondence for Fluentd config file syntax.
Normal Fluentd configuration syntax has the following the list of directives:
- 1.
source
directives determine the input sources - 2.
match
directives determine the output destinations - 3.
filter
directives determine the event processing pipelines - 4.
system
directives set system-wide configuration - 5.
label
directives group the output and filter for internal routing - 6.
worker
directives limit to the specific workers - 7.
@include
directives include other files
In YAML configuration world, we reconstructed them for YAML format.
In YAML syntax, Fluentd will handle the two top level objects:
- 1.
system
The top level object that specifies system settings - 2.
config
Another top level object that defines data pipeline
Under
config
object, Fluentd will handle the following elements:- 1.
source
elements determine the input sources - 2.
match
elements determine the output destinations - 3.
filter
elements determine the event processing pipelines - 4.
label
elements group the output and filter for internal routing
- 1.
!include
defines including rules for other files - 2.
!fluent/s
defines Fluentd string format that is equivalent for double quoted string - 3.
!fluent/json
defines Fluentd JSON format that is used for Hash type object - 4.
$tag
defines tag for output plugin - 5.
$label
defines label routes for input plugin - 6.
$arg
defines directive arg's equivalent objects (e.g. <DIRECTIVE arg>) - 7.
$name
defines label directive name equivalent objects (e.g. <label name>) - 8.
$type
specifies@type
ortype
for instantiating plugin type
Let's actually create a configuration file step by step.
Fluentd input sources are enabled by selecting and configuring the desired input plugins using source elements. Fluentd standard input plugins include
http
and forward
. The http
provides an HTTP endpoint to accept incoming HTTP messages whereas forward
provides a TCP endpoint to accept TCP packets. Of course, it can be both at the same time. You may add multiple source
configurations as required.config:
# Receive events from 24224/tcp
# This is used by log forwarding and the fluent-cat command
- source:
$type: forward
port: 24224
# http://<ip>:9880/myapp.access?json={"event":"data"}
- source:
$type: http
port: 9880
Each source element must include a
$type
object to specify the input plugin to use.You can add new input sources by writing your own plugins. For further information regarding Fluentd input sources, please refer to the Input Plugin Overview article.
The match element looks for events with matching tags and processes them. The most common use of the
match
element is to output events to other systems. For this reason, the plugins that correspond to the match
element are called output plugins. Fluentd standard output plugins include file
and forward
. Let's add those to our configuration file.config:
# Receive events from 24224/tcp
# This is used by log forwarding and the fluent-cat command
- source:
$type: forward
port: 24224
# http://<ip>:9880/myapp.access?json={"event":"data"}
- source:
$type: http
port: 9880
# Match events tagged with "myapp.access" and
# store them to /var/log/fluent/access.%Y-%m-%d
# Of course, you can control how you partition your data
# with the time_slice_format option.
- match:
$tag: myapp.access
$type: file
path: /var/log/fluent/access
Each
match
element must include a $tag
and a $type
parameter. Only events with a $tag
matching the pattern will be sent to the output destination (in the above example, only the events with the tag myapp.access
are matched. See the section below for more advanced usage). The $type
parameter specifies the output plugin to use.Just like input sources, you can add new output destinations by writing custom plugins. For further information regarding Fluentd output destinations, please refer to the Output Plugin Overview article.
The filter element has the same syntax as
match
but filter
could be chained for processing pipeline. Using filters, event flow is like this:Input -> filter 1 -> ... -> filter N -> Output
Let's add standard
record_transformer
filter to match
example.config:
# http://this.host:9880/myapp.access?json={"event":"data"}
- source:
$type: http
port: 9880
- filter:
$tag: myapp.access
$type: record_transformer
record:
host_param: !fluent/s "#{Socket.gethostname}"
- match:
$tag: myapp.access
$type: file
path: /var/log/fluent/access
The received event
{"event":"data"}
goes to record_transformer
filter first. The record_transformer
filter adds host_param
field to the event; and, then the filtered event {"event":"data","host_param":"webserver1"}
goes to the file
output plugin.You can also add new filters by writing your own plugins. For further information regarding Fluentd filter destinations, please refer to the Filter Plugin Overview article.
System-wide configurations are set by
system
element. Most of them are also available via command line options. For example, the following configurations are available:log_level
suppress_repeated_stacktrace
emit_error_log_interval
suppress_config_dump
without_source
process_name
(Only available insystem
element. No fluentd option)
Example:
system:
# equal to -qq option
log_level: error
# equal to --without-source option
without_source:
# ...
If this parameter is set, fluentd supervisor and worker process names are changed.
system:
process_name: fluentd1
With this configuration,
ps
command shows the following result:% ps aux | grep fluentd1
foo 45673 0.4 0.2 2523252 38620 s001 S+ 7:04AM 0:00.44 worker:fluentd1
foo 45647 0.0 0.1 2481260 23700 s001 S+ 7:04AM 0:00.40 supervisor:fluentd1
This feature requires Ruby 2.1 or later.
The label element groups filter and output for internal routing. The
label
reduces the complexity of tag
handling.The
label
element's parameter is a builtin plugin parameter so $name
parameter is needed.Here is a configuration example:
config:
- source:
$type: forward
- source:
$type: tail
$label: '@SYSTEM'
- filter:
$tag: access.**
$type: record_transformer
record:
# ...
- match:
$tag: '**'
$type: elasticsearch
# ...
- label:
$name: '@SYSTEM'
config:
- filter:
$tag: var.log.middleware.**
$type: grep
# ...
- match:
$tag: '**'
$type: s3
# ...
Note that
$label
parameter value should be quoted and label
element should contain config
element within its elements.In this configuration,
forward
events are routed to record_transformer
filter / elasticsearch
output and in_tail
events are routed to grep
filter / s3
output inside @SYSTEM
label.The
$label
parameter is useful for event flow separation without the $tag
route rules.The
@ERROR
label is a builtin label used for error record emitted by plugin's emit_error_event
API.If
$label
with $name: '@ERROR'
is set, the events are routed to this label when the related errors are emitted e.g. the buffer is full or the record is invalid.The
@ROOT
label is a builtin label used for getting root router by plugin's event_emitter_router
API.This label is introduced since v1.14.0 to assign a label back to the default route. For example, timed-out event records are handled by the concat filter can be sent to the default route.
When setting up multiple workers, you can use the worker element to limit plugins to run on specific workers.
This is useful for input and output plugins that do not support multiple workers.
You can use the
$arg N
or $arg N-M
to specify workers. The number is a zero-based worker index.Here is a configuration example:
system:
workers: 4
config:
- source:
$type: sample
tag: test.allworkers
sample: "{\"message\": \"Run with all workers.\"}"
- worker:
$arg: 0
config:
- source:
$type: sample
tag: test.oneworker
sample: "{\"message\": \"Run with only worker-0.\"}"
- worker:
$arg: 0-1
config:
- source:
$type: sample
tag: test.someworkers
sample: "{\"message\": \"Run with worker-0 and worker-1.\"}"
- filter:
$type: record_transformer
$tag: test.**
record:
worker_id: !fluent/s "#{worker_id}"
- match:
$type: stdout
$tag: test.**
The outputs of this config are as follows:
... test.allworkers: {"message":"Run with all workers.","worker_id":"0"}
... test.allworkers: {"message":"Run with all workers.","worker_id":"1"}
... test.allworkers: {"message":"Run with all workers.","worker_id":"2"}
... test.allworkers: {"message":"Run with all workers.","worker_id":"3"}
... test.oneworker: {"message":"Run with only worker-0.","worker_id":"0"}
... test.someworkers: {"message":"Run with worker-0 and worker-1.","worker_id":"0"}
... test.someworkers: {"message":"Run with worker-0 and worker-1.","worker_id":"1"}
The element in separate configuration files can be imported using the !include element:
config:
# Include config files in the ./config.d directory
- !include config.d/*.conf
The
!include
YAML tag supports regular file path, glob pattern, and http URL conventions:config:
# absolute path
- !include /path/to/config.conf
# if using a relative path, the YAML tag will use
# the dirname of this config file to expand the path
- !include extra.conf
# glob match pattern
- !include config.d/*.conf
# http
- !include http://example.com/fluent.conf
Note that for the glob pattern, files are expanded in alphabetical order. If there are
a.conf
and b.conf
then fluentd parses a.conf
first. But, you should not write the configuration that depends on this order. It is so error-prone, therefore, use multiple separate !include
YAML tags for safety.config:
# If you have a.conf, b.conf, ..., z.conf and a.conf / z.conf are important
# This is bad
- !include *.conf
# This is good
- !include a.conf
- !include config.d/*.conf
- !include z.conf
The
!include
YAML tag can be used under sections to share the same parameters:# config file
config:
- match:
$type: forward
# ...
buffer:
$type: file
path: /path/to/buffer/forward
<<: !include /path/to/out_buf_params.yaml
- match:
$type: elasticsearch
# ...
buffer:
$type: file
path: /path/to/buffer/es
<<: !include /path/to/out_buf_params.yaml
# /path/to/out_buf_params.yaml
flush_interval: 5s
total_limit_size: 100m
chunk_limit_size: 1m
Note that, in the middle of element case of
!include
YAML tag usage, users must use <<:
syntax to include other YAML objects successfully.Fluentd tries to match tags in the order that they appear in the config file. So, if you have the following configuration:
config:
# '**' matches all tags. Bad :(
- match:
$tag: '**'
$type: blackhole_plugin
- match:
$tag: myapp.access
$type: file
path: /var/log/fluent/access
then
myapp.access
is never matched. Wider match patterns should be defined after tight match patterns.config:
- match:
$tag: myapp.access
$type: file
path: /var/log/fluent/access
# Capture all unmatched tags. Good :)
- match:
$tag: '**'
$type: blackhole_plugin
Of course, if you use two same patterns, the second
match
is never matched. If you want to send events to multiple outputs, consider out_copy
plugin.The common pitfall is when you put a
filter
element after match
element. It will never work since events never go through the filter for the reason explained above.config:
# You should NOT put this <filter> block after the <match> block below.
# If you do, Fluentd will just emit events without applying the filter.
- filter:
$tag: myapp.access
$type: record_transformer
...
- match:
$tag: myapp.access
$type: file
path: /var/log/fluent/access
When using YAML format syntax in Fluentd configuration, you can use
!fluent/s "#{...}"
to embed arbitrary Ruby code into match patterns. Here is an example:config:
- match:
$tag: fluent/s "app.#{ENV['FLUENTD_TAG']}"
$type: stdout
If you set the environment variable
FLUENTD_TAG
to dev
, this evaluates to app.dev
.The configuration file can be validated without starting the plugins using the
--dry-run
option:$ fluentd --dry-run -c fluent.yaml
This section describes some useful features for the configuration file.
You can write multiline values for
"
quoted string, array and hash values.str_param: "foo # Converts to "foo bar". As NL interpretation will be required for continuos newlines.
bar"
str_param: "foo # Converts to "foo\nbar".
bar"
array_param:
- "a"
- "b"
hash_param: !fluent/json {
"k": "v",
"k1": 10
}
Fluentd assumes
[
or {
is a start of array / hash. So, if you want to set [
or {
started but non-JSON parameter, please use '
or "
.Example # 1: mail plugin
config:
- match:
$tag: '**'
$type: mail
subject: "[CRITICAL] foo's alert system"
Example # 2: map plugin
config:
- match:
$tag:tag
$type: map
map: '[["code." + tag, time, { "code" => record["code"].to_i}], ["time." + tag, time, { "time" => record["time"].to_i}]]'
multi: true
This restriction will be removed with the configuration parser improvement.
You can evaluate the Ruby code with
!fluent/s #{}
in "
quoted string. This is useful for setting machine information e.g. hostname.host_param: !fluent/s "#{Socket.gethostname}" # host_param is actual hostname like `webserver1`.
env_param: !fluent/s "foo-#{ENV['FOO_BAR']}" # NOTE that foo-"#{ENV["FOO_BAR"]}" doesn't work.
In YAML config format,
hostname
and worker_id
shortcuts are also available:host_param: !fluent/s "#{hostname}" # This is same with Socket.gethostname
$id: !fluent/s "out_foo#{worker_id}" # This is same with ENV["SERVERENGINE_WORKER_ID"]
The
worker_id
shortcut is useful under multiple workers. For example, for a separate plugin id, add worker_id
to store the path in s3 to avoid file conflict.Helper methods
use_nil
and use_default
are available:some_param: !fluent/s "#{ENV['FOOBAR'] || use_nil}" # Replace with nil if ENV["FOOBAR"] isn't set
some_param: !fluent/s "#{ENV['FOOBAR'] || use_default}" # Replace with the default value if ENV["FOOBAR"] isn't set
Note that these methods not only replace the embedded Ruby code but the entire string with
nil
or a default value.some_path: !fluent/s "#{use_nil}/some/path" # some_path is nil, not "/some/path"
The backslash
\
is interpreted as an escape character. You need \
for setting "
, , , , \
or several characters in double-quoted string literal.str_param: "foo\nbar" # \n is interpreted as actual LF character
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 1yr ago