You MUST set up your environment according to the steps below before installing Fluentd. Failing to do so will be the cause of many unnecessary problems.
It's HIGHLY recommended that you set up NTP daemon (e.g. chrony, ntpd, etc) on the node to have accurate current timestamp. This is crucial for any production-grade logging services.
For Amazon Web Services users we recommend using AWS hosted NTP server.
Please increase the maximum number of file descriptors. You can check the current number using the
ulimit -n command.
$ ulimit -n65535
If your console shows
1024, it is insufficient. Please add following lines to your
/etc/security/limits.conf file and reboot your machine.
root soft nofile 65536root hard nofile 65536* soft nofile 65536* hard nofile 65536
If you are running fluentd under systemd, the option
LimitNOFILE=65536 can also be used (and if you are using the td-agent package this value is setup by default).
For high load environments consisting of many Fluentd instances, please add these parameters to your
/etc/sysctl.conf file. Please either type
sysctl -p or reboot your node to have the changes take effect.
net.core.somaxconn = 1024net.core.netdev_max_backlog = 5000net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_wmem = 4096 12582912 16777216net.ipv4.tcp_rmem = 4096 12582912 16777216net.ipv4.tcp_max_syn_backlog = 8096net.ipv4.tcp_slow_start_after_idle = 0net.ipv4.tcp_tw_reuse = 1net.ipv4.ip_local_port_range = 10240 65535
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.