This page is for v0.10, not the latest stable version which is v0.12. For the latest stable version of this article, click here.
Table of Contents
Check top command
If Fluentd doesn’t perform as well as you had expected, please check the
top command first. You need to identify which part of your system is the bottleneck (CPU? Memory? Disk I/O? etc).
Avoid extra computations
This is more like a general recommendation, but it’s always better NOT TO HAVE extra computation inside Fluentd. Fluentd is flexible to do quite a bit internally, but adding too much logic to Fluentd’s configuration file makes it difficult to read and maintain, while making it also less robust. The configuration file should be as simple as possible.
Use num_threads parameter
If the destination for your logs is a remote storage or service, adding a
num_threads option will parallelize your outputs (the default is 1). This parameter is available for all output plugins.
<match test> type output_plugin num_threads 8 ... </match>
Use external gzip command for S3/TD
Ruby has GIL (Global Interpreter Lock), which allows only one thread to execute at a time. While I/O tasks can be multiplexed, CPU-intensive tasks will block other jobs. One of the CPU-intensive tasks in Fluentd is compression.
The new version of S3/Treasure Data plugin allows compression outside of the Fluentd process, using gzip. This frees up the Ruby interpreter while allowing Fluentd to process other tasks.
# S3 <match ...> type s3 store_as gzip_command num_threads 8 ... </match> # Treasure Data <match ...> type tdlog use_gzip_command num_threads 8 ... </match>
While not a perfect solution to leverage multiple CPU cores, this can be effective for most Fluentd deployments. As before, you can run this with
num_threads option as well.
The CPU is often the bottleneck for Fluentd instances that handle billions of incoming records. To utilize multiple CPU cores, we recommend using the
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF), originally invented by Treasure Data, Inc. All components are available under the Apache 2 License.