Performance Tuning Single Process

This article describes how to optimize Fluentd's performance within single process. If your traffic is up to 5,000 messages/sec, the following techniques should be enough.
With more traffic, Fluentd tends to be more CPU bound. In such case, please also visit Performance Tuning (Multi-Process) to utilize multiple CPU cores.
Check top command
If Fluentd doesn't perform as well as you had expected, please check the top command first. You need to identify which part of your system is the bottleneck (CPU? Memory? Disk I/O? etc).
Avoid extra computations
This is more like a general recommendation, but it's always better NOT TO HAVE extra computation inside Fluentd. Fluentd is flexible to do quite a bit internally, but adding too much logic to Fluentd's configuration file makes it difficult to read and maintain, while making it also less robust. The configuration file should be as simple as possible.
Use num_threads parameter
If the destination for your logs is a remote storage or service, adding a num_threads option will parallelize your outputs (the default is 1). Using multiple threads can hide the IO/network latency. This parameter is available for all output plugins.
1
<match test>
2
  @type output_plugin
3
  num_threads 8
4
  ...
5
</match>
Copied!
The important point is this option doesn't improve the processing performance, e.g. numerical computation, mutating record, etc.
Use external gzip command for S3/TD
Ruby has GIL (Global Interpreter Lock), which allows only one thread to execute at a time. While I/O tasks can be multiplexed, CPU-intensive tasks will block other jobs. One of the CPU-intensive tasks in Fluentd is compression.
The new version of S3/Treasure Data plugin allows compression outside of the Fluentd process, using gzip. This frees up the Ruby interpreter while allowing Fluentd to process other tasks.
1
# S3
2
<match ...>
3
  @type s3
4
  store_as gzip_command
5
  num_threads 8
6
  ...
7
</match>
8
​
9
# Treasure Data
10
<match ...>
11
  @type tdlog
12
  use_gzip_command
13
  num_threads 8
14
  ...
15
</match>
Copied!
While not a perfect solution to leverage multiple CPU cores, this can be effective for most Fluentd deployments. As before, you can run this with num_threads option as well.
Reduce memory usage
Ruby has several GC parameters to tune GC performance and you can configure these parameters via environment variable(Parameter list is here). To reduce memory usage, set RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR to lower value. RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR is used for full GC trigger and the default is 2.0. Quote from documentation.
1
Do full GC when the number of old objects is more than R * N
2
  where R is this factor and
3
  N is the number of old objects just after last full GC.
Copied!
So default GC behaviour doesn't call full GC until the number of old objects reaches 2.0 * before old objects. This improves the throughput but it grows the total memory usage. This setting is not good for low resource environment, e.g. small container. For such cases, try RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9 or RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=1.2.
See Ruby 2.1 Garbage Collection: ready for production and Watching and Understanding the Ruby 2.1 Garbage Collector at Work article for more detail.
Multi-process plugin
The CPU is often the bottleneck for Fluentd instances that handle billions of incoming records. To utilize multiple CPU cores, we recommend using the in_multiprocess plugin.
​Performance Tuning (Multi Process)​
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.

Performance Tuning Multi Process

Perl

Last modified 2yr ago

Copy link

Contents

Check top command

Avoid extra computations

Use num_threads parameter

Use external gzip command for S3/TD

Reduce memory usage

Multi-process plugin