Fluentd
0.12
0.12
  • Introduction
  • Overview
    • Getting Started
    • Installation
    • Life of a Fluentd event
    • Support
    • FAQ
  • Use Cases
    • Centralized App Logging
    • Monitoring Service Logs
    • Data Analytics
    • Connecting to Data Storages
    • Stream Processing
    • Windows Event Collection
    • IoT Data Logger
  • Configuration
    • Config File Syntax
    • Routing Examples
    • Recipes
  • Deployment
    • Logging
    • Monitoring
    • Signals
    • RPC
    • High Availability Config
    • Failure Scenarios
    • Performance Tuning
    • Plugin Management
    • Trouble Shooting
    • Secure Forwarding
    • Fluentd UI
    • Command Line Option
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Input Plugins
    • tail
    • forward
    • secure_forward
    • udp
    • tcp
    • http
    • unix
    • syslog
    • exec
    • scribe
    • multiprocess
    • dummy
    • Others
  • Output Plugins
    • file
    • s3
    • kafka
    • forward
    • secure_forward
    • exec
    • exec_filter
    • copy
    • geoip
    • roundrobin
    • stdout
    • null
    • webhdfs
    • splunk
    • mongo
    • mongo_replset
    • relabel
    • rewrite_tag_filter
    • Others
  • Buffer Plugins
    • memory
    • file
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
  • Developer
    • Plugin Development
    • Community
    • Mailing List
    • Source Code
    • Bug Tracking
    • ChangeLog
    • Logo
  • Articles
    • Store Apache Logs into MongoDB
    • Apache To Riak
    • Store Apache Logs into Amazon S3
    • Before Install
    • Cep Norikra
    • Collect Glusterfs Logs
    • Common Log Formats
    • Docker Logging Efk Compose
    • Docker Logging
    • Filter Modify Apache
    • Forwarding Over Ssl
    • Free Alternative To Splunk By Fluentd
    • Data Collection to Hadoop (HDFS)
    • Data Analytics with Treasure Data
    • Install By Chef
    • Install By Deb
    • Install By Dmg
    • Install By Gem
    • Install By Rpm
    • Install From Source
    • Install On Beanstalk
    • Install On Heroku
    • Java
    • Kinesis Stream
    • Kubernetes Fluentd
    • Monitoring by Prometheus
    • Monitoring by Rest Api
    • Nodejs
    • Performance Tuning Multi Process
    • Performance Tuning Single Process
    • Perl
    • Php
    • Python
    • Quickstart
    • Raspberrypi Cloud Data Logger
    • Recipe Apache Logs To Elasticsearch
    • Recipe Apache Logs To Mongo
    • Recipe Apache Logs To S3
    • Recipe Apache Logs To Treasure Data
    • Recipe Cloudstack To Mongodb
    • Recipe Csv To Elasticsearch
    • Recipe Csv To Mongo
    • Recipe Csv To S3
    • Recipe Csv To Treasure Data
    • Recipe Http Rest Api To Elasticsearch
    • Recipe Http Rest Api To Mongo
    • Recipe Http Rest Api To S3
    • Recipe Http Rest Api To Treasure Data
    • Recipe Json To Elasticsearch
    • Recipe Json To Mongo
    • Recipe Json To S3
    • Recipe Json To Treasure Data
    • Recipe Nginx To Elasticsearch
    • Recipe Nginx To Mongo
    • Recipe Nginx To S3
    • Recipe Nginx To Treasure Data
    • Recipe Syslog To Elasticsearch
    • Recipe Syslog To Mongo
    • Recipe Syslog To S3
    • Recipe Syslog To Treasure Data
    • Recipe Tsv To Elasticsearch
    • Recipe Tsv To Mongo
    • Recipe Tsv To S3
    • Recipe Tsv To Treasure Data
    • Ruby
    • Scala
    • Splunk Like Grep And Alert Email
Powered by GitBook
On this page
  • Look at Logs
  • Turn on Verbose Logging
  • rpm/deb
  • gem
  • Dump fluentd internal information
  • High CPU usage issue
  • Check uncaught logs

Was this helpful?

  1. Deployment

Trouble Shooting

Look at Logs

If things aren't happening as expected, please first look at your logs. For td-agent (rpm/deb), the logs are located at /var/log/td-agent/td-agent.log.

Turn on Verbose Logging

You can get more information about the logs if verbose logging is turned on. Please follow the steps below.

rpm/deb

  1. Open the service setting file for td-agent (see below for its file

    path).

  2. Add -vv to TD_AGENT_OPTIONS.

  3. Restart td-agent.

    # Add the following settings to:
    # - /etc/sysconfig/td-agent (CentOS/RedHat)
    # - /etc/default/td-agent (Debian/Ubuntu)
    TD_AGENT_OPTIONS="-vv"

Note: The environment variable TD_AGENT_OPTIONS has been introduced in td-agent v2.2.1. If you are using an older version of td-agent, you need to edit /etc/init.d/td-agent and insert -vv options to TD_AGENT_ARGS or DAEMON_ARGS manually.

gem

Please add -vv to your command line.

$ fluentd .. -vv

Dump fluentd internal information

High CPU usage issue

If fluentd suddenly hits unexpected high CPU usage problem, there are several reasons:

  • a plugin has a race condition or similar bug

  • dependent gems have a bug

  • regular expression with broken data

  • system calls has a bug, e.g. inotify with lots of files

Check uncaught logs

You sometimes hit unexpected shutdown with non-zero exit status like below.

2016-01-01 00:00:00 +0800 [info]: starting fluentd-0.12.28
2016-01-01 00:00:00 +0800 [info]: reading config file path="/etc/td-agent/td-agent.conf"
[...snip...]
2016-01-01 00:00:02 +0800 [info]: process finished code=6

If the problem happens inside ruby, e.g. segmentation fault, C extension bug, etc, you can't get entire log when fluentd process is daemonized. For example, td-agent launches fluentd with --daemon option. In td-agent case, you can get entire log using following command to simulate /etc/init.d/td-agent start without daemonize.

$ sudo LD_PRELOAD=/opt/td-agent/embedded/lib/libjemalloc.so /usr/sbin/td-agent -c /etc/td-agent/td-agent.conf --user td-agent --group td-agent
PreviousPlugin ManagementNextSecure Forwarding

Last updated 5 years ago

Was this helpful?

Fluentd uses for dumping fluentd internal information to local file, e.g. thread dump, object allocation and etc. If you have a problem with fluentd like process hang, please send SIGCONT to fluentd parent and child processes.

In such cases, you can use perf tool on recent Linux to investigate the problem. See page. If you want to know which call causes the problem, is useful.

If this article is incorrect or outdated, or omits critical information, please . is a open source project under . All components are available under the Apache 2 License.

sigdump
Linux perf Examples
pid2line.rb
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)