Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Log Level
  • Global Logs
  • By Command Line Option
  • By Config File
  • Per Plugin Log
  • Log Format
  • Suppress log/stacktrace messages
  • ignore_repeated_log_interval
  • ignore_same_log_interval
  • Output to Log File
  • By Command Line Option
  • By Config File
  • Log Rotation Setting
  • --log-rotate-age AGE
  • --log-rotate-size BYTES
  • Capture Fluentd logs
  • Case 1: Send Fluentd Logs to Monitoring Service
  • Case 2: Use Aggregation/Monitoring Server
  • Deprecate Top-Level Match

Was this helpful?

  1. Deployment

Logging

This article describes the Fluentd logging mechanism.

Fluentd has two logging layers: global and per plugin. Different log levels can be set for global logging and plugin level logging.

Log Level

Here is the list of supported levels in increasing order of verbosity:

  • fatal

  • error

  • warn

  • info

  • debug

  • trace

The default log level is info, and Fluentd outputs info, warn, error and fatal logs by default.

Global Logs

Global logging is used by Fluentd core and plugins that do not set their own log levels. The global log level can be adjusted up or down.

By Command Line Option

Increase Verbosity Level

The -v option sets the verbosity to debug while the -vv option sets the verbosity to trace.

$ fluentd -v  ... # debug log level
$ fluentd -vv ... # trace log level

These options are useful for debugging purposes.

Decrease Verbosity Level

The -q option sets the verbosity to warn while the -qq option sets the verbosity to error:

$ fluentd -q  ... # warn log level
$ fluentd -qq ... # error log level

By Config File

You can also configure the logging level in <system> section:

# same as -qq command line option

<system>
  log_level error
</system>

Per Plugin Log

The @log_level option sets different levels of logging for each plugin. It can be set in each plugin's configuration file.

<source>
  @type tail
  @log_level debug
  path /var/log/data.log
  # ...
</source>

<source>
  @type http
  @log_level fatal
</source>

If you do not specify the @log_level parameter, the plugin will use the global log level.

Log Format

Following format are supported:

  • text (default)

  • json

The format can be configured through <log> directive under <system>:

<system>
  <log>
    format json
    time_format %Y-%m-%d
  </log>
</system>

With this setting, the following log line:

2017-07-27 06:44:54 +0900 [info]: #0 fluentd worker is now running worker=0

is changed to:

{"time":"2017-07-27","level":"info","message":"fluentd worker is now running worker=0","worker_id":0}

Suppress log/stacktrace messages

Fluentd provides two parameters to suppress log/stacktrace messages

  • ignore_repeated_log_interval (since v1.10.2)

  • ignore_same_log_interval (since v1.11.3)

ignore_repeated_log_interval

<system>
  ignore_repeated_log_interval 60s
</system>

Under high loaded environment, output destination sometimes becomes unstable and it causes lots of same log message. This parameter mitigates such situation.

ignore_same_log_interval

<system>
  ignore_same_log_interval 60s
</system>

This is similar to ignore_repeated_log_interval but covers more usecases. For example, if the plugin generates several log messages in one action, logs are not repeated:

# Retry generates several type messages. ignore_repeated_log_interval can't suppress these messages
def write(chunk)
  # process 1...
  log.error "message1"
  # process 2...
  log.error "message2"
end

ignore_same_log_interval resolves these cases.

Output to Log File

By Command Line Option

By default, Fluentd outputs to the standard output. Use -o command line option to specify the file instead:

$ fluentd -o /path/to/log_file

By Config File

Since v1.18.0, You can also configure the log file path using the <log> directive under <system>:

<system>
  <log>
    path /path/to/log_file
  </log>
</system>

Log Rotation Setting

By default, Fluentd does not rotate log files. You can configure this behavior via system-config after v1.13.0.

It can be configured through <log> directive under <system>:

<system>
  <log>
    rotate_age 5
    rotate_size 1048576
  </log>
</system>

NOTE: You can omit one of these 2 options to use the default value, but if you omit both of them, log rotation is disabled. Actually, an external library manages these default values, resulting in this complication.

You can use command-line options too (mainly for before v1.13.0):

--log-rotate-age AGE

AGE is an integer or a string:

  • integer: Generations to keep rotated log files.

  • string: frequency of rotation. (Supported: daily, weekly, monthly)

NOTE: When --log-rotate-age is specified on Windows, log files are separated into log-supervisor-0.log, log-0.log, ..., log-N.log where N is generation - 1 due to the system limitation. Windows does not permit delete and rename files simultaneously owned by another process.

--log-rotate-size BYTES

The byte size to rotate log files. This is applied when --log-rotate-age is specified with integer:

Here is an example:

$ fluentd -c fluent.conf --log-rotate-age 5 --log-rotate-size 104857600

NOTE: When --log-rotate-size is specified on Windows, log files are separated into log-supervisor-0.log, log-0.log, ..., log-N.log where N is generation - 1 due to the system limitation. Windows does not permit delete and rename files simultaneously owned by another process.

Capture Fluentd logs

Fluentd marks its own logs with the fluent tag. You can process Fluentd logs by using <match fluent.**>(Of course, ** captures other logs) in <label @FLUENT_LOG>. If you define <label @FLUENT_LOG> in your configuration, then Fluentd will send its own logs to this label. This is useful for monitoring Fluentd logs.

For example, if you have the following configuration:

# omit other source / match

<label @FLUENT_LOG>
  <match fluent.*>
    @type stdout
  </match>
</label>

Then, Fluentd outputs fluent.info logs to stdout like this:

2014-02-27 00:00:00 +0900 [info]: shutting down fluentd
2014-02-27 00:00:01 +0900 fluent.info: {"message":"shutting down fluentd"} # by <match fluent.*>
2014-02-27 00:00:01 +0900 [info]: process finished code = 0

Case 1: Send Fluentd Logs to Monitoring Service

You can send Fluentd logs to a monitoring service by plugins e.g. datadog, sentry, irc, etc.

# Add hostname for identifying the server

<label @FLUENT_LOG>
  <filter fluent.*>
    @type record_transformer
    <record>
      host "#{Socket.gethostname}"
    </record>
  </filter>

  <match fluent.*>
    @type monitoring_plugin
    # ...
  </match>
</label>

Case 2: Use Aggregation/Monitoring Server

Leaf Server Example:

# Add hostname for identifying the server and tag to filter by log level

<label @FLUENT_LOG>
  # If you want to capture only error events, use 'fluent.error' instead.

  <filter fluent.*>
    @type record_transformer
    <record>
      host "#{Socket.gethostname}"
      original_tag ${tag}
    </record>
  </filter>

  <match fluent.*>
    @type forward
    <server>
      # Monitoring server parameters
    </server>
  </match>
</label>

Monitoring Server Example:

<source>
  @type forward
</source>

# Ignore trace, debug and info log. Of course, you can use strict matching
# like `<filter fluent.{warn,error,fatal}>` without grep filter.

<filter fluent.*>
  @type grep
  <regexp>
    key original_tag
    pattern fluent.(warn|error|fatal)
  </regexp>
</filter>

# your notification setup. This example uses irc plugin
<match fluent.*>
  @type irc
  host irc.domain
  channel notify
  message notice: %s [%s] @%s %s
  out_keys original_tag,time,host,message
</match>

# Unlike v0.12, if `<label @FLUENT_LOG>` is defined,
# `<match fluent.*>` in root is not used for log capturing.

<label @FLUENT_LOG>
  # Monitoring server's internal log
</label>

If an error occurs, you will get a notification message in your Slack notify channel:

01:01  fluentd: [11:10:24] notice: fluent.warn [2014/02/27 01:00:00] @leaf.server.domain detached forwarding server 'server.name'

Deprecate Top-Level Match

PreviousSystem ConfigurationNextSignals

Last updated 3 months ago

Was this helpful?

For example, in order to debug and to suppress all but fatal log messages for , their respective @log_level options should be set as follows:

You need to specify or options explicitly to enable log rotation.

You can use to send Fluentd logs to a monitoring server. The monitoring server can then filter and send the logs to your notification system e.g. chat, irc, etc.

You can still use without <label @FLUENT_LOG> but this feature is deprecated. This feature will be removed in fluentd v2.

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

in_tail
in_http
out_forward
v0.12 way
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
rotate_age
rotate_size