Fluentd
Search…
Logging
This article describes Fluentd's logging mechanism.
Fluentd has two log layers: global and per plugin. Different log levels can be set for global logging and plugin level logging.
Log Level
Shown below is the list of supported values, in increasing order of verbosity:
fatalerrorwarninfodebugtrace
The default log level is
info, and Fluentd outputs info, warn, error and fatal logs by default.Global Logs
Global logging is used by Fluentd core and plugins that don't set their own log levels. The global log level can be adjusted up or down.
By Command Line Option
Increase Verbosity Level
The
-v option sets the verbosity to debug while the -vv option sets the verbosity to trace.1
$ fluentd -v ... # debug level
2
$ fluentd -vv ... # trace level
Copied!
These options are useful for debugging purposes.
Decrease Verbosity Level
The
-q option sets the verbosity to warn while the -qq option sets the verbosity to error.1
$ fluentd -q ... # warn level
2
$ fluentd -qq ... # error level
Copied!
By Config File
You can also change the logging level with
<system> section in the config file like below.1
<system>
2
# equal to -qq option
3
log_level error
4
</system>
Copied!
Per Plugin Log
The
log_level option sets different levels of logging for each plugin. It can be set in each plugin's configuration file.1
<source>
2
@type tail
3
@log_level debug
4
path /var/log/data.log
5
...
6
</source>
7
<source>
8
@type http
9
@log_level fatal
10
</source>
Copied!
If you don't specify the
log_level parameter, the plugin will use the global log level. Some plugins haven't supported per-plugin logging yet. The logging section of the Plugin Development article explains how to update such plugins to support the new log level system.Suppress repeated stacktrace
Fluentd can suppress same stacktrace with
--suppress-repeated-stacktrace. For example, if you pass --suppress-repeated-stacktrace to fluentd:1
2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:154:rescue in emit_stream: emit transaction failed error_class = RuntimeError error = #<RuntimeError: syslog>
2
2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/out_stdout.rb:43:in `emit'
3
[snip]
4
2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/in_object_space.rb:63:in `run'
5
2013-12-04 15:05:53 +0900 [error]: plugin/in_object_space.rb:113:rescue in on_timer: object space failed to emit error = "foo.bar" error_class = "RuntimeError" tag = "foo" record = "{ ...}"
6
2013-12-04 15:05:55 +0900 [warn]: fluent/engine.rb:154:rescue in emit_stream: emit transaction failed error_class = RuntimeError error = #<RuntimeError: syslog>
7
2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/o/2.0.0/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run'
8
[snip]
Copied!
logs are changed to:
1
2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:154:rescue in emit_stream: emit transaction failed error_class = RuntimeError error = #<RuntimeError: syslog>
2
2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/o/2.0.0/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run'
3
[snip]
4
2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/in_object_space.rb:63:in `run'
5
2013-12-04 15:05:53 +0900 [error]: plugin/in_object_space.rb:113:rescue in on_timer: object space failed to emit error = "foo.bar" error_class = "RuntimeError" tag = "foo" record = "{ ...}"
6
2013-12-04 15:05:55 +0900 [warn]: fluent/engine.rb:154:rescue in emit_stream: emit transaction failed error_class = RuntimeError error = #<RuntimeError: syslog>
7
2013-12-04 15:05:55 +0900 [warn]: plugin/in_object_space.rb:111:on_timer: suppressed same stacktrace
Copied!
Same stacktrace is replaced with
suppressed same stacktrace message until other stacktrace is received.Output to log file
Fluentd outputs logs to
STDOUT by default. To output to a file instead, please specify the -o option.1
$ fluentd -o /path/to/log_file
Copied!
Fluentd doesn't support log rotation yet.
Capture Fluentd logs
Fluentd marks its own logs with the
fluent tag. You can process Fluentd logs by using <match fluent.**> or <match **>(Of course, ** captures other logs). If you define <match fluent.**> in your configuration, then Fluentd will send its own logs to this match destination. This is useful for monitoring Fluentd logs.For example, if you have the following
<match fluent.**>:1
# omit other source / match
2
<match fluent.**>
3
@type stdout
4
</match>
Copied!
then Fluentd outputs
fluent.info logs to stdout like below:1
2014-02-27 00:00:00 +0900 [info]: shutting down fluentd
2
2014-02-27 00:00:01 +0900 fluent.info: {"message":"shutting down fluentd"} # by <match fluent.**>
3
2014-02-27 00:00:01 +0900 [info]: process finished code = 0
Copied!
Case1: Send Fluentd logs to monitoring service
You can send Fluentd logs to a monitoring service by plugins, e.g. datadog, sentry, irc, etc.
1
# Add hostname for identifying the server
2
<filter fluent.**>
3
@type record_transformer
4
<record>
5
host "#{Socket.gethostname}"
6
</record>
7
</filter>
8
​
9
<match fluent.**>
10
@type monitoring_plugin
11
# parameters...
12
</match>
Copied!
Case2: Use aggregation/monitoring server
You can use out_forward to send Fluentd logs to a monitoring server. The monitoring server can then filter and send the logs to your notification system: chat, irc, etc.
Leaf server example:
1
# Add hostname for identifying the server and tag to filter by log level
2
<filter fluent.**>
3
@type record_transformer
4
<record>
5
host "#{Socket.gethostname}"
6
original_tag ${tag}
7
</record>
8
</filter>
9
​
10
<match fluent.**>
11
@type forward
12
<server>
13
# Monitoring server parameters
14
</server>
15
</match>
Copied!
Monitoring server example:
1
<source>
2
@type forward
3
label @FLUENTD_INTERNAL_LOG
4
</source>
5
​
6
<label @FLUENTD_INTERNAL_LOG>
7
# Ignore trace, debug and info log
8
<filter fluent.**>
9
@type grep
10
regexp1 original_tag fluent.(warn|error|fatal)
11
</filter>
12
​
13
<match fluent.**>
14
# your notification setup. This example uses irc plugin
15
@type irc
16
host irc.domain
17
channel notify
18
message notice: %s [%s] @%s %s
19
out_keys original_tag,time,host,message
20
</match>
21
</label>
Copied!
If an error occurs, you will get a notification message in your irc
notify channel.1
01:01 fluentd: [11:10:24] notice: fluent.warn [2014/02/27 01:00:00] @leaf.server.domain detached forwarding server 'server.name'
Copied!
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 2yr ago