Fluentd has nine (9) types of plugins:
This article gives an overview of the Output Plugin.
Fluentd v1.0 output plugins have three (3) buffering and flushing modes:
- Non-Buffered mode does not buffer data and write out resultsimmediately.
- Synchronous Buffered mode has "staged" buffer chunks (a chunk is acollection of events) and a queue of chunks, and its behavior can becontrolled by
<buffer>section (See the diagram below).
- Asynchronous Buffered mode also has "stage" and "queue", butthe output plugin will not commit writing chunks in methodssynchronously, but commit them later.
Fluentd v1.0 Plugin API Overview
Output plugins can support all the modes, but may support just one of these modes. Fluentd chooses appropriate mode automatically if there are no
<buffer>sections in the configuration. If the users specify
<buffer>section for the output plugins that do not support buffering, Fluentd will raise configuration errors.
Output plugins in v1 can control keys of buffer chunking by configurations, dynamically. Users can configure buffer chunk keys as time (any unit specified by user), tag and any key name of records. Output plugin will split events into chunks: events in a chunk have the same values for chunk keys. The output plugin's buffer behavior (if any) is defined by a separate Buffer plugin. Different buffer plugins can be chosen for each output plugin.
See this list of available plugins to find out more about other Output plugins:
Fluentd v0.12 uses only
<match>section for both the configuration parameters of output and buffer plugins. Fluentd v1.0 uses
<buffer>subsection to write parameters for buffering, flushing and retrying.
<match>sections are used only for the output plugin itself.
Example of v1.0 output plugin configuration:
For Fluentd v0.12, configuration parameters for buffer plugins are written in the same section:
immediateflushes just after event arrives
The interval between buffer chunk flushes.
The number of threads to flush the buffer.
Seconds to sleep between checks for buffer flushes in flush threads.
Seconds to sleep between flushes when many buffer chunks are queued.
Seconds of timeout for buffer chunks to be committed by plugins later.
The threshold for chunk flush performance check.
Note that the parameter type is
If chunk flush takes longer time than this threshold, fluentd logs warning message like this:
2016-12-19 12:00:00 +0000 [warn]: buffer flush took longer time than slow_flush_log_threshold: elapsed_time=15.0031226690043695 slow_flush_log_threshold=10.0 plugin_id="foo"
Controls the buffer behavior when the queue becomes full.
throw_exception(default)This mode throws the
BufferOverflowErrorexception to the input plugin. How
BufferOverflowErroris handled depends on the input plugins, e.g. tail input stops reading new lines. This action is suitable for streaming.
blockThis mode stops input plugin thread until buffer full issue is resolved. This action is good for batch-like use-cases. This is mainly for
in_tailplugin. Other input plugins, e.g. socket-based plugin, don't assume this action.We do not recommend using
blockaction to avoid
BufferOverflowError. Please consider improving destination settings to resolve
@ERRORlabel for routing overflowed events to another backup destination (or
retry_limit). If you hit
BufferOverflowErrorfrequently, it means your destination capacity is insufficient for your traffic.
drop_oldest_chunkThis mode drops the oldest chunks. This mode is useful for monitoring system destinations. For monitoring, newer events are important than older.
If the bottom chunk write out fails, it will remain in the queue and Fluentd will retry after waiting for several seconds (
retry_wait). If the retry limit has not been disabled (
false) and the retry count exceeds the specified limit (
retry_max_times), all chunks in the queue are discarded. The retry wait time doubles each time (1.0sec, 2.0sec, 4.0sec, ...) until
retry_max_intervalis reached. If the queue length exceeds the specified limit (
queue_limit_length), new events are rejected.
Writing out the bottom chunk is considered to be a failure if
Output#try_writemethod throws an exception.
The retry timings of
Specifies how to wait for the next retry to flush buffer.
true, plugin will ignore
retry_max_timesoptions and retry flushing forever.
The maximum seconds to retry to flush while failing, until the plugin discards the buffer chunks. If the next retry is going to exceed this time limit, the last retry will be made at exactly this time limit.
The maximum number of times to retry to flush while failing. If
retry_timeoutis the default, the number is 18 with exponential backoff.
The ratio of
retry_timeoutto switch to use secondary while failing.
Seconds to wait before the next retry to flush, or constant factor of exponential backoff.
The base number of exponential backoff for retries.
The maximum interval (seconds) for exponential backoff between retries while failing.
true, the output plugin will retry after a randomized interval not to do burst retries.
bufferedmode, the user can specify
<secondary>with any output plugin in
<match>configuration. If plugins continue to fail writing buffer chunks and exceeds the timeout threshold for retries, then output plugins will delegate the writing of the buffer chunk to the secondary plugin.
<secondary>is useful for backup when destination servers are unavailable, e.g.
mongo, etc. We strongly recommend
This example sends logs to Elasticsearch using a file buffer
/var/log/td-agent/buffer/elasticsearch, and any failure will be sent to
my.logsfor file names:
<secondary>plugin receives the primary's buffer chunk directly. So, you need to check if your secondary plugin works with the primary setting.
The retry timings of
retry_timeout: 100swith the secondary.
The retry timings of
retry_max_times: 10with the secondary.