Before installing Fluentd, make sure that your environment is properly set up to avoid any inconsistencies at a later stage.
Follow these recommendations:
Set Up NTP
Increase the Maximum Number of File Descriptors
Optimize the Network Kernel Parameters
It is highly recommended that you set up an NTP daemon (e.g.
ntpd, etc.) on the node to have an accurate current timestamp. This is crucial for all the production-grade logging services.
For Amazon Web Services users, we recommend using AWS-hosted NTP server.
Increase the maximum number of file descriptors. You can check the existing configuration using the
ulimit -n command:
$ ulimit -n65535
If your console shows
1024, it is insufficient. Please add the following lines to your
/etc/security/limits.conf file and reboot your machine:
root soft nofile 65536root hard nofile 65536* soft nofile 65536* hard nofile 65536
If you are running fluentd under
systemd, the option
LimitNOFILE=65536 can also be used. And, if you are using the
td-agent package, this value is set up by default.
For high load environments with many Fluentd instances, add the following configuration to your
net.core.somaxconn = 1024net.core.netdev_max_backlog = 5000net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_wmem = 4096 12582912 16777216net.ipv4.tcp_rmem = 4096 12582912 16777216net.ipv4.tcp_max_syn_backlog = 8096net.ipv4.tcp_slow_start_after_idle = 0net.ipv4.tcp_tw_reuse = 1net.ipv4.ip_local_port_range = 10240 65535
sysctl -p command or reboot your node for the changes to take effect.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.