Versions | v0.14 (td-agent3) | v0.12 (td-agent2) | v0.10 (td-agent1)

This page is for v0.14, not the latest stable version which is v0.12. For the latest stable version of this article, click here.


Output Plugin Overview

Fluentd has 6 types of plugins: Input, Parser, Filter, Output, Formatter, Storage and Buffer. This article gives an overview of Output Plugin.

Table of Contents

Overview

Fluentd v0.14 output plugins have 3 modes about buffering and flushing.

  • Non-Buffered mode doesn’t buffer data and write out results immediately.
  • Synchronous Buffered mode has “staged” buffer chunks (a chunk is a collection of events) and a queue of chunks, and its behavior can be controlled by <buffer> section (See the diagram below).
  • Asynchronous Buffered mode also has “stage” and “queue”, but output plugin will not commit writing chunks in methods synchronously, but commit later.

Fluentd v0.14 Plugin API Overview

Output plugins can support all modes, but may support just one of these modes. Fluentd choose appropriate mode automatically if there are no <buffer> sections in configuration. If users specify <buffer> section for output plugins which doesn’t support buffering, Fluentd will stop with configuration errors.

Output plugins in v0.14 can control keys of buffer chunking by configurations, dynamically. Users can configure buffer chunk keys as time (any unit specified by user), tag and any key name of records. Output plugin will split events into chunks: events in a chunk have same values for chunk keys. The output plugin’s buffer behavior (if any) is defined by a separate Buffer plugin. Different buffer plugins can be chosen for each output plugin.

List of Output Plugins

Other Plugins

Please refer to this list of available plugins to find out about other Output plugins.

Difference between v0.14 and v0.12

Fluentd v0.12 uses only <match> section for both of configuration parameters of output plugin and buffer plugin. Fluentd v0.14 uses <buffer> subsection to write parameters for buffering, flushing and retrying. <match> sections are used only for output plugin itself.

Example of v0.14 output plugin configuration:

<match myservice_name>
  @type file
  path /my/data/access.${tag}.%Y-%m-%d.%H%M.log
  <buffer tag,time>
    @type file
    path /my/buffer/myservice
    timekey     60m
    timekey_wait 1m
  </buffer>
</source>

For Fluentd v0.12, configuration parameters for buffer plugins were written in same section:

<match myservice_name>
  @type file
  path /my/data/access.myservice_name.*.log
  buffer_type file
  buffer_path /my/buffer/myservice/access.myservice_name.*.log
  time_slice_format %Y-%m-%d.%H%M
  time_slice_wait   1m
</source>

Buffering/Retrying Parameters

TODO: list parameters available in <buffer> section

Control Flushing

TODO: This description is copied from buffer overview of v0.12, and must be updated with v0.14 features.

If queue is empty when chunks are pushed into queue, queued chunk will be flushed immediately.

When the top chunk exceeds the specified size or time limit (buffer_chunk_limit and flush_interval, respectively), a new empty chunk is pushed to the top of the queue. The bottom chunk is written out immediately when new chunk is pushed.

If the bottom chunk write out fails, it will remain in the queue and Fluentd will retry after waiting several seconds (retry_wait). If the retry limit has not been disabled (disable_retry_limit is false) and the retry count exceeds the specified limit (retry_limit), the chunk is trashed. The retry wait time doubles each time (1.0sec, 2.0sec, 4.0sec, …) until max_retry_wait is reached. If the queue length exceeds the specified limit (buffer_queue_limit), new events are rejected.

writing out the bottom chunk is considered to be a failure if "BufferedOutput#write" method throws an exception.

Control Retrying

For other configuration parameters available in <buffer> sections, see Buffer plugin overview and each plugins.

Secondary Output

In buffered mode, the user can specify <secondary> with any output plugin in <match> configuration. If plugins continue to fail writing buffer chunks and exceeds the timeout threshold for retries, then output plugins will delegate to write the buffer chunk to secondary plugin.

<secondary> is useful for backup when destination servers are unavailable, e.g. forward, mongo and other plugins. We strongly recommend out_file plugin for <secondary>.

TODO: add examples for secondary output

Last updated: 2016-06-13 06:11:23 UTC

Versions | v0.14 (td-agent3) | v0.12 (td-agent2) | v0.10 (td-agent1)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF), originally invented by Treasure Data, Inc. All components are available under the Apache 2 License.

Interested in the Fluentd Newsletters?