Fluentd
0.12
0.12
  • Introduction
  • Overview
    • Getting Started
    • Installation
    • Life of a Fluentd event
    • Support
    • FAQ
  • Use Cases
    • Centralized App Logging
    • Monitoring Service Logs
    • Data Analytics
    • Connecting to Data Storages
    • Stream Processing
    • Windows Event Collection
    • IoT Data Logger
  • Configuration
    • Config File Syntax
    • Routing Examples
    • Recipes
  • Deployment
    • Logging
    • Monitoring
    • Signals
    • RPC
    • High Availability Config
    • Failure Scenarios
    • Performance Tuning
    • Plugin Management
    • Trouble Shooting
    • Secure Forwarding
    • Fluentd UI
    • Command Line Option
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Input Plugins
    • tail
    • forward
    • secure_forward
    • udp
    • tcp
    • http
    • unix
    • syslog
    • exec
    • scribe
    • multiprocess
    • dummy
    • Others
  • Output Plugins
    • file
    • s3
    • kafka
    • forward
    • secure_forward
    • exec
    • exec_filter
    • copy
    • geoip
    • roundrobin
    • stdout
    • null
    • webhdfs
    • splunk
    • mongo
    • mongo_replset
    • relabel
    • rewrite_tag_filter
    • Others
  • Buffer Plugins
    • memory
    • file
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
  • Developer
    • Plugin Development
    • Community
    • Mailing List
    • Source Code
    • Bug Tracking
    • ChangeLog
    • Logo
  • Articles
    • Store Apache Logs into MongoDB
    • Apache To Riak
    • Store Apache Logs into Amazon S3
    • Before Install
    • Cep Norikra
    • Collect Glusterfs Logs
    • Common Log Formats
    • Docker Logging Efk Compose
    • Docker Logging
    • Filter Modify Apache
    • Forwarding Over Ssl
    • Free Alternative To Splunk By Fluentd
    • Data Collection to Hadoop (HDFS)
    • Data Analytics with Treasure Data
    • Install By Chef
    • Install By Deb
    • Install By Dmg
    • Install By Gem
    • Install By Rpm
    • Install From Source
    • Install On Beanstalk
    • Install On Heroku
    • Java
    • Kinesis Stream
    • Kubernetes Fluentd
    • Monitoring by Prometheus
    • Monitoring by Rest Api
    • Nodejs
    • Performance Tuning Multi Process
    • Performance Tuning Single Process
    • Perl
    • Php
    • Python
    • Quickstart
    • Raspberrypi Cloud Data Logger
    • Recipe Apache Logs To Elasticsearch
    • Recipe Apache Logs To Mongo
    • Recipe Apache Logs To S3
    • Recipe Apache Logs To Treasure Data
    • Recipe Cloudstack To Mongodb
    • Recipe Csv To Elasticsearch
    • Recipe Csv To Mongo
    • Recipe Csv To S3
    • Recipe Csv To Treasure Data
    • Recipe Http Rest Api To Elasticsearch
    • Recipe Http Rest Api To Mongo
    • Recipe Http Rest Api To S3
    • Recipe Http Rest Api To Treasure Data
    • Recipe Json To Elasticsearch
    • Recipe Json To Mongo
    • Recipe Json To S3
    • Recipe Json To Treasure Data
    • Recipe Nginx To Elasticsearch
    • Recipe Nginx To Mongo
    • Recipe Nginx To S3
    • Recipe Nginx To Treasure Data
    • Recipe Syslog To Elasticsearch
    • Recipe Syslog To Mongo
    • Recipe Syslog To S3
    • Recipe Syslog To Treasure Data
    • Recipe Tsv To Elasticsearch
    • Recipe Tsv To Mongo
    • Recipe Tsv To S3
    • Recipe Tsv To Treasure Data
    • Ruby
    • Scala
    • Splunk Like Grep And Alert Email
Powered by GitBook
On this page
  • 2-Tier Process Topology
  • 1st Tier: Input Processes
  • 2nd Tier: Output Processes
  • Reasons behind this Design
  • Example Configuration
  • Files
  • How to add input processes?
  • How to add output processes?
  • How many processes should we have?
  • Fluentd v1.0 or Later

Was this helpful?

  1. Articles

Performance Tuning Multi Process

PreviousNodejsNextPerformance Tuning Single Process

Last updated 5 years ago

Was this helpful?

This article desdrivbes how to optimize Fluentd's performance with in_multiprocess plugin. With high traffic, Fluentd tends to be more CPU bound.

However with Ruby MRI's GVL () limitation, Fluentd can use only single CPU core. With in_multiprocessm Fluentd can fully utilize multiple CPU cores to handle more requests.

Before you have multi-process configuration, please make sure you have done .

2-Tier Process Topology

In multi-process environment, we recommend to have the following 2-tier process topology within the same server.

1st Tier: Input Processes

These processes will be in charge for input (typicall either in_tail or other input plugins) and its parsing. The input processes will transfer all the records to 2nd tier output processes, by out_forward.

In the config, it will be recommended to use shorter flush_interval (e.g. 1s) with smaller buffer_chunk_size (e.g. 1MB) to immediately forward incoming data to 2nd tier output processes.

2nd Tier: Output Processes

These process will be in charge for filter and output to your favorite destinations. The output processes accept incoming data from input processes, by in_forward.

The output processes are shared across all input processes, by out_forward's load balancing mechanism. In this way, you can simply add more output processes to handle more capacity across all inputs.

Reasons behind this Design

There are coupole of reasons why we recommend this design.

Focus

Each process can use nearly 100% of its CPU with much smaller focus (either input or output). For example, in_tail can now focus on tailing and parsing, rather than sharing the CPU power for filtering and output.

Scale

To handle more volume, you can simply add more input and output processes. in_forward and out_forward automatically distribute the load across the processes automatically.

Robust

If any of the process goes down, the supervisor process will automatically relaunch the process. Also we recommend to use buf_file for both input and output processes, to simply prevent losing the data.

For example, even if one of the output processes die, the data gets buffered and routed to different output processes automatically. Also crashed processes will be automaticlaly relaunched by supervisor process.

Example Configuration

This git repository contains the fully functional multi-process settings for Fluentd.

You can simply test with td-agent, or directly use Ruby to launch Fluentd.

$ bundle install
$ bundle exec fluentd -c fluentd.conf

Files

fluentd.conf (Supervisor)

This config uses in_multiprocess to launch both input and output processes.

in_*.conf (1st Tier Input Processes)

The example config contains 3 input processes: in_tcp1.conf, in_tcp2.conf, and in_udp1.conf.

These input processes will focus on receiving the data via either TCP or UDP, and sipmly forward to 2nd tier output processes.

out_*.conf (2nd Tier Output Processes)

The example config contains 4 output processes: out1.conf, out2.conf, out3.conf, out4.conf.

These output processes receive data from input processes, apply filters, and output to the destinations.

./tmpl/ directory has the config generation script, in case you'd like to increase the processes.

$ bash gen_out.sh

monitoring_*.conf (Monitoring)

How to add input processes?

Please add in_xyz.conf. Please make sure you have <match> section, to forward incoming data to 2nd tier output processes. Finally, register the process in fluentd.conf.

How to add output processes?

Please add out_N.conf, and register the process in both options_forward.conf and fluentd.conf which list up all the output processes.

How many processes should we have?

We recommend to have approximately similar number of total processes with your CPU cores.

When you perform the benchmark, please carefully look at which processes are the bottleneck. Please add the processes depending on its resource usages.

Fluentd v1.0 or Later

Fluentd v1.0 or later has native multi-process support. We recommend you to upgrade to simplify the config file if possible.

These files are used to collect metrics from Fluentd, and exposing the metrics to monitoring system. The repository also has ./prometheus directory, which contains example prometheus.yml configuration.

If this article is incorrect or outdated, or omits critical information, please . is a open source project under . All components are available under the Apache 2 License.

Fluentd Multi-Process Example Configuration (Githib)
Prometheus
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
Global VM Lock
all the optimization you can do with single process