Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Introduction and Getting Started
  • My logs have stopped sending data to my backend
  • My logs are filled with BufferOverflow Error
  • I'm missing data in my backend destination
  • My logs are being parsed incorrectly

Was this helpful?

Troubleshooting Guide

PreviousPlugin Helper: Service DiscoveryNextAppendix

Last updated 1 year ago

Was this helpful?

Introduction and Getting Started

Fluentd has thousands of plugins and tons of configuration options to read from various different data sources. However, this flexibility can also make it difficult to troubleshoot. The following Troubleshooting guide goes over a few steps to follow in case of issues and how to solve them.

If you need additional help we also recommend the following options

Community Support

Enterprise Support

My logs have stopped sending data to my backend

This is probably one of the most common issues users face with Fluentd and either not seeing data in their configured backend or data has stopped flowing. There are a few possibilities for why Fluentd has stopped sending logs to your backend destination

Fluentd is waiting for the retry interval In the case that the backend is unreachable (network failure or application log rejection) Fluentd automatically engages in a retry process that follows an to avoid causing a denial of service event. As the backoff sequence implies it can grow relatively quickly from a few seconds to a couple of hours. In the log file, you will see next retry interval at XYZ which can help you diagnose this issue.

Suggestions:

  • Add retry_max_interval to the output plugin configuration section to cap the max amount of time to wait until attempting a retry

  • If you are using the RPC endpoint you can call flush buffers manually ()

  • Restarting the Daemon can immediately restart the interval NOTE: if you are using memory buffers vs. on-disk buffers this could result in data loss

Fluentd cannot reach your backend In some cases, the network environment can suddenly change not allowing Fluentd to connect with a previously accessible point

Suggestions:

  • Try running Fluentd manually via command line with manual input to ensure that you can reach the end destination

Datasource is no longer flowing in or has completed Another possible reason for Fluentd to stop sending data is that there is no longer new data flowing into the input plugin that Fluentd is configured to use. An example of this can be that a log file has been rotated and Fluentd is configured to tail a specific log file

Suggestions:

  • Check that the input configuration is correct and uses * where appropriate.

My logs are filled with BufferOverflow Error

Suggestions:

  • Check that flush_interval is low enough that you are continuously flushing the buffer as you are reading data. For example if you are reading 10,000 events / second make sure you are not flushing data every hour otherwise your buffer can quickly fill up

  • Increase workers and flush_thread_count. if you have excessive messages per second and Fluentd is failing to keep adjusting these two settings will increase Fluentd's resource utilization but hopefully allow the application to keep up with the required throughput

  • Change buffer type from memory to file. If you are running into this problem you might have exceeded the default total memory buffer size of 512MB. Fluentd uses a small default to prevent excessive memory usage, however can be configured to use filesystem for lower resource usage (memory) and more resiliency through restarts. The default sizes for total_limit_size file buffers are also much larger (64GB file vs. 512 MB memory)

  • Additionally, you can manually Increase buffer sizing by increasing the following parameter: total_limit_size as well as changing the maximum size of chunks chunk_limit_size.

I'm missing data in my backend destination

Missing data in your backend destination can also be a wildcard error that needs to be checked in multiple places. These can include the configuration of your input / filter / output plugins, the buffering system, and if you are potentially awaiting a retry. Here are a few suggestions you can run through

Suggestions:

  • Check that configuration is set correctly. For example, if you are reading from a file and you do not see records from the beginning of the file check if read_from_head is true

  • Check your log file for BufferOverflowError. You may be sending data at high throughput and data might be dropped due to a number of issues. Follow troubleshooting steps in BufferOverflowError section

  • Check existing buffers that have not flushed yet. If you are using a file buffer check if there is data that is present in buffer_path and has not been flushed

My logs are being parsed incorrectly

One of Fluentd's big strengths is it's ability to parse logs into a standardized format based on custom formats or well-known formats. However writing regular expressions can be hard to validate and ensure that the proper fields are working. Here are a few tips I recommend

Suggestions:

  • Check multi-line logs have proper configuration settings. Multi-line logs need to have specific settings set in the parser to work properly, make sure these are all set.

If using a network input plugin check that data is flowing or using fluent-cat to mimic a message being sent:

Fluentd output plugins generally contain buffers that are stored in either memory (default) or on disk. These buffers are configurable with both how many chunks are allowed as well as the number of chunks to allow to be in the total buffer. When the buffer is full then the plugin will automatically return BufferOverflowError which in the case of continuously writing data (E.g. Syslog, Network logs) will pop up in the Fluentd log file. , and

Use online tools such as and with a sample of the log file as well as the regex. This will give you a clear example of what the end match groups will look like as well as allow you to validate Ruby timeformat.

Check existing parsers for well-known formats you may be able to use one of the parsers that Fluentd already includes . Note: you can also checkout for additional examples as both projects use Ruby regular expressions

Fluent Slack Channel
Fluentd GitHub Discussions
Fluentd GitHub Issues
Enterprise Providers
exponential backoff sequence
https://docs.fluentd.org/deployment/rpc
further reading
Further Reading on Buffer Docs
Further Reading on Buffering / Retrying parameters
Rubular.com
fluentular
https://docs.fluentd.org/parser
Fluent Bit parsers