Output Plugins
Fluentd has 7 types of plugins: Input, Parser, Filter, Output, Formatter, Storage and Buffer. This article gives an overview of Output Plugin.
Fluentd v1.0 output plugins have 3 modes about buffering and flushing.
Non-Buffered mode doesn't buffer data and write out results
immediately.
Synchronous Buffered mode has "staged" buffer chunks (a chunk is a
collection of events) and a queue of chunks, and its behavior can be
controlled by
<buffer>section (See the diagram below).Asynchronous Buffered mode also has "stage" and "queue", but
output plugin will not commit writing chunks in methods
synchronously, but commit later.
Output plugins can support all modes, but may support just one of these modes. Fluentd choose appropriate mode automatically if there are no <buffer> sections in configuration. If users specify <buffer> section for output plugins which doesn't support buffering, Fluentd will stop with configuration errors.
Output plugins in v1 can control keys of buffer chunking by configurations, dynamically. Users can configure buffer chunk keys as time (any unit specified by user), tag and any key name of records. Output plugin will split events into chunks: events in a chunk have same values for chunk keys. The output plugin's buffer behavior (if any) is defined by a separate Buffer plugin. Different buffer plugins can be chosen for each output plugin.
​out_copy​
​out_null​
​out_roundrobin​
​out_stdout​
​out_exec_filter​
​out_forward​
​out_mongo or out_mongo_replset​
​out_exec​
​out_file​
​out_s3​
​out_webhdfs​
Please refer to this list of available plugins to find out about other Output plugins.
​others​
Fluentd v0.12 uses only <match> section for both of configuration parameters of output plugin and buffer plugin. Fluentd v1.0 uses <buffer> subsection to write parameters for buffering, flushing and retrying. <match> sections are used only for output plugin itself.
Example of v1.0 output plugin configuration:
<match myservice_name>@type filepath /my/data/access.${tag}.%Y-%m-%d.%H%M.log<buffer tag,time>@type filepath /my/buffer/myservicetimekey 60mtimekey_wait 1m</buffer></match>
For Fluentd v0.12, configuration parameters for buffer plugins were written in same section:
<match myservice_name>@type filepath /my/data/access.myservice_name.*.logbuffer_type filebuffer_path /my/buffer/myservice/access.myservice_name.*.logtime_slice_format %Y-%m-%d.%H%Mtime_slice_wait 1m</match>
See buffer section in Compat Parameters Plugin Helper API for parameter name changes between v1 and v0.12.
See Buffer section configurations.
See Buffer Plugin Overview for the basic behaviour of buffer.
The default is default. Supported types are default, lazy, interval or immediate
How to enqueue chunks to be flushed. "interval" flushes per flush_interval, "immediate" flushes just after event arrival.
the default is 60
The interval between buffer chunk flushes.
The default is 1.
The number of threads to flush the buffer.
The default is 1.0
Seconds to sleep between checks for buffer flushes in flush threads.
The default is 1.0
Seconds to sleep between flushes when many buffer chunks are queued.
The default is 60.
Seconds of timeout for buffer chunks to be committed by plugins later
The threshold for chunk flush performance check. The default value is 20.0 seconds. Note that parameter type is float, not time.
If chunk flush takes longer time than this threshold, fluentd logs warning message like below:
2016-12-19 12:00:00 +0000 [warn]: buffer flush took longer time than slow_flush_log_threshold: elapsed_time=15.0031226690043695 slow_flush_log_threshold=10.0 plugin_id="foo"
Control the buffer behaviour when the queue becomes full. 3 modes are supported:
throw_exception
This is default mode. This mode raises BufferOverflowError exception to input plugin. How handle BufferOverflowError depends on input plugins, e.g. tail input stops reading new lines, forward input returns an error to forward output. This action fits for streaming manner.
block
This mode stops input plugin thread until buffer full is resolved. This action is good for batch-like use-case.
We don't recommend to use block action to avoid BufferOverflowError. Please consider improving destination setting to resolve BufferOverflowError or use @ERROR label for routing overflowed events to another backup destination(or secondary with lower retry_limit). If you hit BufferOverflowError frequently, it means your destination capacity is insufficient for your traffic.
drop_oldest_chunk
This mode drops oldest chunks. This mode is useful for monitoring system destinations. For monitoring, newer events are important than older.
If the bottom chunk write out fails, it will remain in the queue and Fluentd will retry after waiting several seconds (retry_wait). If the retry limit has not been disabled (retry_forever is false) and the retry count exceeds the specified limit (retry_max_times), all chunks in the queue are discarded. The retry wait time doubles each time (1.0sec, 2.0sec, 4.0sec, ...) until retry_max_interval is reached. If the queue length exceeds the specified limit (queue_limit_length), new events are rejected.
writing out the bottom chunk is considered to be a failure if Output#write or Output#try_write method throws an exception.
The default is exponential_backoff.
How to wait next retry to flush buffer. Supported types are exponential_backoff or periodic
The default is false.
If true, plugin will ignore retry_timeout and retry_max_times options and retry flushing forever.
The default is 72 hours.
The maximum seconds to retry to flush while failing, until plugin discards buffer chunks.
The default is nil
The maximum number of times to retry to flush while failing. If retry_timeout is the default, the number is 17 with exponential backoff.
The default is 0.8
The ratio of retry_timeout to switch to use secondary while failing.
The default is 1
Seconds to wait before next retry to flush, or constant factor of exponential backoff.
The default is 2
The base number of exponential backoff for retries.
The default is nil.
The maximum interval seconds for exponential backoff between retries while failing.
The default is true.
If true, output plugin will retry after randomized interval not to do burst retries.
For other configuration parameters available in <buffer> sections, see Buffer plugin overview and each plugins.
In buffered mode, the user can specify <secondary> with any output plugin in <match> configuration. If plugins continue to fail writing buffer chunks and exceeds the timeout threshold for retries, then output plugins will delegate to write the buffer chunk to secondary plugin.
<secondary> is useful for backup when destination servers are unavailable, e.g. forward, mongo and other plugins. We strongly recommend out_secondary_file plugin for <secondary>.
The example below sends logs to Elasticsearch using a file buffer /var/log/td-agent/buffer/elasticsearch, and any failures will be sent to /var/log/td-agent/error/ using my.logs for file names.
<match my.logs>@type elasticsearchhost localhostport 9200logstash_format true<buffer>@type filepath /var/log/td-agent/buffer/elasticsearch</buffer><secondary>@type filepath /var/log/td-agent/error/my.logs</secondary></match>
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
