multiprocess
By default, Fluentd only uses a single CPU core on the system. The
in_multiprocess
Input plugin enables Fluentd to use multiple CPU cores by spawning multiple child processes. One Fluentd user is using this plugin to handle 10+ billion records / day.in_multiprocess
is NOT included in td-agent by default. td-agent users must install fluent-plugin-multiprocess manually.- Fluentd:
fluent-gem install fluent-plugin-multiprocess
- td-agent v2:
/usr/sbin/td-agent-gem install fluent-plugin-multiprocess
- td-agent v1:
/usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-multiprocess
<source>
@type multiprocess
<process>
cmdline -c /etc/fluent/fluentd_child1.conf --log /var/log/fluent/fluentd_child1.log
sleep_before_start 1s
sleep_before_shutdown 5s
</process>
<process>
cmdline -c /etc/fluent/fluentd_child2.conf --log /var/log/fluent/fluentd_child2.log
sleep_before_start 1s
sleep_before_shutdown 5s
</process>
<process>
cmdline -c /etc/fluent/fluentd_child3.conf --log /var/log/fluent/fluentd_child3.log
sleep_before_start 1s
sleep_before_shutdown 5s
</process>
</source>
Especially for daemonized fluentd (ex: td-agent), `--log` option MUST be specified to put logs of child processes.
The value must be
multiprocess
.The interval to send the signal to gracefully shut down the process (default: 2sec).
The increment time, when graceful shutdown fails (default: 3sec).
The timeout, to identify the failure of graceful shutdown (default: 60sec).
The
process
section sets the command line arguments of a child process. This plugin creates one child process for each section.The
cmdline
option is required in a sectionThis parameter sets the wait time before starting the process (default: 0sec).
This parameter sets the wait time before shutting down the process (default: 0sec).
The
log_level
option allows the user to set different levels of logging for each plugin. The supported log levels are: fatal
, error
, warn
, info
, debug
, and trace
.If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 3yr ago