Versions | v0.14 (td-agent3) | v0.12 (td-agent2) | v0.10 (td-agent1)

Before Installing Fluentd

You MUST set up your environment according to the steps below before installing Fluentd. Failing to do so will be the cause of many unnecessary problems.

Table of Contents

Set Up NTP

It’s HIGHLY recommended that you set up ntpd on the node to prevent invalid timestamps in your logs.

Increase Max # of File Descriptors

Please increase the maximum number of file descriptors. You can check the current number using the ulimit -n command.

$ ulimit -n
65535

If your console shows 1024, it is insufficient. Please add following lines to your /etc/security/limits.conf file and reboot your machine.

root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536

This 65536 is safer value for large deployment. The required number of file descriptors depends on your fluentd plugins and setting. Here are the estimation examples:

  • in_tail: The number of watching files
  • in_forward: The number of incoming access
  • buf_file: The number of buffer chunks. It is configured via buffer parameters
  • output: The used file descriptors are less than others. Temporary file for upload, connection pooling in the client library, etc

Optimize Network Kernel Parameters

For high load environments consisting of many Fluentd instances, please add these parameters to your /etc/sysctl.conf file. Please either type sysctl -p or reboot your node to have the changes take effect. If your environment doesn’t have a problem with TCP_WAIT, then these changes are not needed.

net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240    65535
Last updated: 2017-02-24 18:44:45 UTC

Versions | v0.14 (td-agent3) | v0.12 (td-agent2) | v0.10 (td-agent1)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF), originally invented by Treasure Data, Inc. All components are available under the Apache 2 License.

Interested in the Fluentd Newsletters?