Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Look at Logs
  • Turn on Verbose Logging
  • Use Directive in Configuration File
  • td-agent
  • calyptia-fluentd
  • gem
  • Dump fluentd's Internal Information
  • High CPU Usage Issue
  • Check Uncaught Logs

Was this helpful?

  1. Deployment

Trouble Shooting

PreviousPlugin ManagementNextFluentd UI

Last updated 2 years ago

Was this helpful?

Look at Logs

If things are not happening as expected, please first look at your logs. For td-agent (rpm/deb), the logs are located at /var/log/td-agent/td-agent.log.

Turn on Verbose Logging

You can get more information about the logs if verbose logging is turned on. Please follow the steps below:

Use Directive in Configuration File

Set log_level trace.

See the article also.

td-agent

td-agent systemd with User's Unit File

Put your unit file into /etc/systemd/system/td-agent. This overwrites the existing behavior of /usr/lib/systemd/system/td-agent.

[Service]
ExecStart=...existing options... -vv

td-agent init.d with Environment Variable

  1. edit /etc/init.d/td-agent

  2. add -vv to TD_AGENT_OPTIONS

  3. restart td-agent

    # at /etc/init.d/td-agent
    ...
    TD_AGENT_OPTIONS="... -vv"
    ...

This approach does not work on the systemd environment because systemd hooks the init.d's startup routine and ignores the other options.

calyptia-fluentd

calyptia-fluentd systemd with User's Unit File

Put your unit file into /etc/systemd/system/calyptia-fluentd.service. This overwrites the existing behavior of /usr/lib/systemd/system/calyptia-fluentd.service.

[Service]
ExecStart=...existing options... -vv

calyptia-fluentd is only provided systemd enabled platforms on Linux.

gem

Please add -vv to your command line:

$ fluentd .. -vv

Dump fluentd's Internal Information

High CPU Usage Issue

If fluentd suddenly hits unexpected high CPU usage problem, there are several reasons:

  • a plugin has a race condition or similar bug

  • dependent gems have a bug

  • regular expression with broken data

  • system calls has a bug, e.g. inotify with lots of files

Check Uncaught Logs

You sometimes hit unexpected shutdown with non-zero exit status like this:

2016-01-01 00:00:00 +0800 [info]: starting fluentd-0.12.28
2016-01-01 00:00:00 +0800 [info]: reading config file path="/etc/td-agent/td-agent.conf"
[...snip...]
2016-01-01 00:00:02 +0800 [info]: process finished code=6

If the problem happens inside Ruby e.g. segmentation fault, C extension bug, etc., you cannot get the complete log when fluentd process is daemonized. For example, td-agent launches fluentd with --daemon option. In td-agent case, you can get the complete log with following command to simulate /etc/init.d/td-agent start without daemonizing (run in the foreground):

$ sudo LD_PRELOAD=/opt/td-agent/embedded/lib/libjemalloc.so /usr/sbin/td-agent -c /etc/td-agent/td-agent.conf --user td-agent --group td-agent

Fluentd uses for dumping fluentd internal information to a local file, e.g. thread dump, object allocation, etc. If you have a problem with fluentd like process hang, please send SIGCONT to fluentd parent and child processes.

On Windows, you can use .

In such cases, you can use perf tool on recent Linux to investigate the problem. See page. If you want to know which call causes the problem, is useful.

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

SIGDUMP
Linux perf Examples
pid2line.rb
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
fluent-ctl
logging