This page is for v0.10, not the latest stable version which is v0.12. For the latest stable version of this article, click here.
Before Installing Fluentd
You MUST set up your environment according to the steps below before installing Fluentd. Failing to do so will be the cause of many unnecessary problems.
Table of Contents
Set Up NTP
It’s HIGHLY recommended that you set up ntpd on the node to prevent invalid timestamps in your logs.
Increase Max # of File Descriptors
Please increase the maximum number of file descriptors. You can check the current number using the
ulimit -n command.
$ ulimit -n 65535
If your console shows
1024, it is insufficient. Please add following lines to your
/etc/security/limits.conf file and reboot your machine.
root soft nofile 65536 root hard nofile 65536 * soft nofile 65536 * hard nofile 65536
65536 is safer value for large deployment. The required number of file descriptors depends on your fluentd plugins and setting. Here are the estimation examples:
- in_tail: The number of watching files
- in_forward: The number of incoming access
- buf_file: The number of buffer chunks. It is configured via buffer parameters
- output: The used file descriptors are less than others. Temporary file for upload, connection pooling in the client library, etc
Optimize Network Kernel Parameters
For high load environments consisting of many Fluentd instances, please add these parameters to your
/etc/sysctl.conf file. Please either type
sysctl -p or reboot your node to have the changes take effect. If your environment doesn’t have a problem with TCP_WAIT, then these changes are not needed.
net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.ip_local_port_range = 10240 65535
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF), originally invented by Treasure Data, Inc. All components are available under the Apache 2 License.