Versions | v0.12 (td-agent2) | v0.10 (td-agent1)

This page is for v0.10, not the latest stable version which is v0.12. For the latest stable version of this article, click here.


Performance Tuning

Table of Contents

Check top command

If Fluentd doesn’t perform as well as you had expected, please check the top command first. You need to identify which part of your system is the bottleneck (CPU? Memory? Disk I/O? etc).

Avoid extra computations

This is more like a general recommendation, but it’s always better NOT TO HAVE extra computation inside Fluentd. Fluentd is flexible to do quite a bit internally, but adding too much logic to Fluentd’s configuration file makes it difficult to read and maintain, while making it also less robust. The configuration file should be as simple as possible.

Use num_threads parameter

If the destination for your logs is a remote storage or service, adding a num_threads option will parallelize your outputs (the default is 1). This parameter is available for all output plugins.

<match test>
  type output_plugin
  num_threads 8
  ...
</match>

Use external gzip command for S3/TD

Ruby has GIL (Global Interpreter Lock), which allows only one thread to execute at a time. While I/O tasks can be multiplexed, CPU-intensive tasks will block other jobs. One of the CPU-intensive tasks in Fluentd is compression.

The new version of S3/Treasure Data plugin allows compression outside of the Fluentd process, using gzip. This frees up the Ruby interpreter while allowing Fluentd to process other tasks.

# S3
<match ...>
  type s3
  store_as gzip_command
  num_threads 8
  ...
</match>

# Treasure Data
<match ...>
  type tdlog
  use_gzip_command
  num_threads 8
  ...
</match>

While not a perfect solution to leverage multiple CPU cores, this can be effective for most Fluentd deployments. As before, you can run this with num_threads option as well.

Multi-process plugin

The CPU is often the bottleneck for Fluentd instances that handle billions of incoming records. To utilize multiple CPU cores, we recommend using the in_multiprocess plugin.

Last updated: 2016-06-06 04:47:13 UTC

Versions | v0.12 (td-agent2) | v0.10 (td-agent1)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.

Interested in the Fluentd Newsletters?