Logging

Search…
Logging
This article describes the Fluentd logging mechanism.
Fluentd has two logging layers: global and per plugin. Different log levels can be set for global logging and plugin level logging.
Log Level
Here is the list of supported levels in increasing order of verbosity:
fatal
error
warn
info
debug
trace
The default log level is info, and Fluentd outputs info, warn, error and fatal logs by default.
Global Logs
Global logging is used by Fluentd core and plugins that do not set their own log levels. The global log level can be adjusted up or down.
By Command Line Option
Increase Verbosity Level
The -v option sets the verbosity to debug while the -vv option sets the verbosity to trace.
1
$ fluentd -v  ... # debug log level
2
$ fluentd -vv ... # trace log level
Copied!
These options are useful for debugging purposes.
Decrease Verbosity Level
The -q option sets the verbosity to warn while the -qq option sets the verbosity to error:
1
$ fluentd -q  ... # warn log level
2
$ fluentd -qq ... # error log level
Copied!
By Config File
You can also configure the logging level in <system> section:
1
# same as -qq command line option
2
​
3
<system>
4
  log_level error
5
</system>
Copied!
Per Plugin Log
The @log_level option sets different levels of logging for each plugin. It can be set in each plugin's configuration file.
For example, in order to debug in_tail and to suppress all but fatal log messages for in_http, their respective @log_level options should be set as follows:
1
<source>
2
  @type tail
3
  @log_level debug
4
  path /var/log/data.log
5
  # ...
6
</source>
7
​
8
<source>
9
  @type http
10
  @log_level fatal
11
</source>
Copied!
If you do not specify the @log_level parameter, the plugin will use the global log level.
Log Format
Following format are supported:
text (default)
json
The format can be configured through <log> directive under <system>:
1
<system>
2
  <log>
3
    format json
4
    time_format %Y-%m-%d
5
  </log>
6
</system>
Copied!
With this setting, the following log line:
1
2017-07-27 06:44:54 +0900 [info]: #0 fluentd worker is now running worker=0
Copied!
is changed to:
1
{"time":"2017-07-27","level":"info","message":"fluentd worker is now running worker=0","worker_id":0}
Copied!
Suppress log/stacktrace messages
Fluentd provides two parameters to suppress log/stacktrace messages
ignore_repeated_log_interval (since v1.10.2)
ignore_same_log_interval (since v1.11.3)
ignore_repeated_log_interval
1
<system>
2
  ignore_repeated_log_interval 60s
3
</system>
Copied!
Under high loaded environment, output destination sometimes becomes unstable and it causes lots of same log message. This parameter mitigates such situation.
ignore_same_log_interval
1
<system>
2
  ignore_same_log_interval 60s
3
</system>
Copied!
This is similar to ignore_repeated_log_inteval but covers more usecases. For example, if the plugin generates several log messages in one action, logs are not repeated:
1
# Retry generates several type messages. ignore_repeated_log_interval can't suppress these messages
2
def write(chunk)
3
  # process 1...
4
  log.error "message1"
5
  # process 2...
6
  log.error "message2"
7
end
Copied!
ignore_same_log_interval resolves these cases.
Output to Log File
By default, Fluentd outputs to the standard output. Use -o command line option to specify the file instead:
1
$ fluentd -o /path/to/log_file
Copied!
Log Rotation Setting
By default, Fluentd does not rotate log files. You can configure this behavior via command-line options:
--log-rotate-age AGE
AGE is an integer or a string:
integer: Generations to keep rotated log files.
string: frequency of rotation. (Supported: daily, weekly, monthly)
NOTE: When --log-rotate-age is specified on Windows, log files are separated into log-supervisor-0.log, log-0.log, ..., log-N.log where N is generation - 1 due to the system limitation. Windows does not permit delete and rename files simultaneously owned by another process.
--log-rotate-size BYTES
The byte size to rotate log files. This is applied when --log-rotate-age is specified with integer:
Here is an example:
1
$ fluentd -c fluent.conf --log-rotate-age 5 --log-rotate-size 104857600
Copied!
NOTE: When --log-rotate-size is specified on Windows, log files are separated into log-supervisor-0.log, log-0.log, ..., log-N.log where N is generation - 1 due to the system limitation. Windows does not permit delete and rename files simultaneously owned by another process.
Capture Fluentd logs
Fluentd marks its own logs with the fluent tag. You can process Fluentd logs by using <match fluent.**>(Of course, ** captures other logs) in <label @FLUENT_LOG>. If you define <label @FLUENT_LOG> in your configuration, then Fluentd will send its own logs to this label. This is useful for monitoring Fluentd logs.
For example, if you have the following configuration:
1
# omit other source / match
2
​
3
<label @FLUENT_LOG>
4
  <match fluent.*>
5
    @type stdout
6
  </match>
7
</label>
Copied!
Then, Fluentd outputs fluent.info logs to stdout like this:
1
2014-02-27 00:00:00 +0900 [info]: shutting down fluentd
2
2014-02-27 00:00:01 +0900 fluent.info: {"message":"shutting down fluentd"} # by <match fluent.*>
3
2014-02-27 00:00:01 +0900 [info]: process finished code = 0
Copied!
Case 1: Send Fluentd Logs to Monitoring Service
You can send Fluentd logs to a monitoring service by plugins e.g. datadog, sentry, irc, etc.
1
# Add hostname for identifying the server
2
​
3
<label @FLUENT_LOG>
4
  <filter fluent.*>
5
    @type record_transformer
6
    <record>
7
      host "#{Socket.gethostname}"
8
    </record>
9
  </filter>
10
​
11
  <match fluent.*>
12
    @type monitoring_plugin
13
    # ...
14
  </match>
15
</label>
Copied!
Case 2: Use Aggregation/Monitoring Server
You can use out_forward to send Fluentd logs to a monitoring server. The monitoring server can then filter and send the logs to your notification system e.g. chat, irc, etc.
Leaf Server Example:
1
# Add hostname for identifying the server and tag to filter by log level
2
​
3
<label @FLUENT_LOG>
4
  # If you want to capture only error events, use 'fluent.error' instead.
5
​
6
  <filter fluent.*>
7
    @type record_transformer
8
    <record>
9
      host "#{Socket.gethostname}"
10
      original_tag ${tag}
11
    </record>
12
  </filter>
13
​
14
  <match fluent.*>
15
    @type forward
16
    <server>
17
      # Monitoring server parameters
18
    </server>
19
  </match>
20
</label>
Copied!
Monitoring Server Example:
1
<source>
2
  @type forward
3
</source>
4
​
5
# Ignore trace, debug and info log. Of course, you can use strict matching
6
# like `<filter fluent.{warn,error,fatal}>` without grep filter.
7
​
8
<filter fluent.*>
9
  @type grep
10
  <regexp>
11
    key original_tag
12
    pattern fluent.(warn|error|fatal)
13
  </regexp>
14
</filter>
15
​
16
# your notification setup. This example uses irc plugin
17
<match fluent.*>
18
  @type irc
19
  host irc.domain
20
  channel notify
21
  message notice: %s [%s] @%s %s
22
  out_keys original_tag,time,host,message
23
</match>
24
​
25
# Unlike v0.12, if `<label @FLUENT_LOG>` is defined,
26
# `<match fluent.*>` in root is not used for log capturing.
27
​
28
<label @FLUENT_LOG>
29
  # Monitoring server's internal log
30
</label>
Copied!
If an error occurs, you will get a notification message in your Slack notify channel:
1
01:01  fluentd: [11:10:24] notice: fluent.warn [2014/02/27 01:00:00] @leaf.server.domain detached forwarding server 'server.name'
Copied!
Deprecate Top-Level Match
You can still use v0.12 way without <label @FLUENT_LOG> but this feature is deprecated. This feature will be removed in fluentd v2.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.