Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Overview
  • List of Output Plugins
  • Other Plugins
  • Difference between v1.0 and v0.12
  • Buffering/Retrying Parameters
  • Control Flushing
  • Control Retrying
  • Secondary Output

Was this helpful?

Output Plugins

Previouswindows_eventlogNextfile

Last updated 2 years ago

Was this helpful?

Fluentd has nine (9) types of plugins:

This article gives an overview of the Output Plugin.

Overview

Fluentd v1.0 output plugins have three (3) buffering and flushing modes:

  • Non-Buffered mode does not buffer data and write out results

    immediately.

  • Synchronous Buffered mode has "staged" buffer chunks (a chunk is a

    collection of events) and a queue of chunks, and its behavior can be

    controlled by <buffer> section (See the diagram below).

  • Asynchronous Buffered mode also has "stage" and "queue", but

    the output plugin will not commit writing chunks in methods

    synchronously, but commit them later.

Output plugins can support all the modes, but may support just one of these modes. Fluentd chooses appropriate mode automatically if there are no <buffer> sections in the configuration. If the users specify <buffer> section for the output plugins that do not support buffering, Fluentd will raise configuration errors.

List of Output Plugins

Other Plugins

See this list of available plugins to find out more about other Output plugins:

Difference between v1.0 and v0.12

Fluentd v0.12 uses only <match> section for both the configuration parameters of output and buffer plugins. Fluentd v1.0 uses <buffer> subsection to write parameters for buffering, flushing and retrying. <match> sections are used only for the output plugin itself.

Example of v1.0 output plugin configuration:

<match myservice_name>
  @type file
  path /my/data/access.${tag}.%Y-%m-%d.%H%M.log
  <buffer tag,time>
    @type file
    path /my/buffer/myservice
    timekey 60m
    timekey_wait 1m
  </buffer>
</match>

For Fluentd v0.12, configuration parameters for buffer plugins are written in the same section:

<match myservice_name>
  @type file
  path /my/data/access.myservice_name.*.log
  buffer_type file
  buffer_path /my/buffer/myservice/access.myservice_name.*.log
  time_slice_format %Y-%m-%d.%H%M
  time_slice_wait 1m
</match>

Buffering/Retrying Parameters

Control Flushing

flush_mode

Default: default

Supported types: default, lazy, interval, immediate

  • interval flushes per flush_interval

  • immediate flushes just after event arrives

flush_interval

The interval between buffer chunk flushes.

Default: 60

flush_thread_count

The number of threads to flush the buffer.

Default: 1

flush_thread_interval

Seconds to sleep between checks for buffer flushes in flush threads.

Default: 1.0

flush_thread_burst_interval

Seconds to sleep between flushes when many buffer chunks are queued.

Default: 1.0

delayed_commit_timeout

Seconds of timeout for buffer chunks to be committed by plugins later.

Default: 60

slow_flush_log_threshold

The threshold for chunk flush performance check.

Default: 20.0 (seconds)

Note that the parameter type is float, not time.

If chunk flush takes longer time than this threshold, fluentd logs warning message like this:

2016-12-19 12:00:00 +0000 [warn]: buffer flush took longer time than slow_flush_log_threshold: elapsed_time=15.0031226690043695 slow_flush_log_threshold=10.0 plugin_id="foo"

overflow_action

Controls the buffer behavior when the queue becomes full.

Supported modes:

  • throw_exception (default)

    This mode throws theBufferOverflowError exception to the input plugin. How BufferOverflowError is handled depends on the input plugins, e.g. tail input stops reading new lines. This action is suitable for streaming.

  • block

    This mode stops input plugin thread until buffer full issue is resolved. This action is good for batch-like use-cases. This is mainly for in_tail plugin. Other input plugins, e.g. socket-based plugin, don't assume this action.

    We do not recommend using block action to avoid BufferOverflowError. Please consider improving destination settings to resolve BufferOverflowError or use @ERROR label for routing overflowed events to another backup destination (or secondary with lower retry_limit). If you hit BufferOverflowError frequently, it means your destination capacity is insufficient for your traffic.

  • drop_oldest_chunk

    This mode drops the oldest chunks. This mode is useful for monitoring system destinations. For monitoring, newer events are important than older.

Control Retrying

If the bottom chunk write out fails, it will remain in the queue and Fluentd will retry after waiting for several seconds (retry_wait). If the retry limit has not been disabled (retry_forever is false) and the retry count exceeds the specified limit (retry_max_times), all chunks in the queue are discarded. The retry wait time doubles each time (1.0sec, 2.0sec, 4.0sec, ...) until retry_max_interval is reached. If the queue length exceeds the specified limit (queue_limit_length), new events are rejected.

Writing out the bottom chunk is considered to be a failure if Output#write or Output#try_write method throws an exception.

The retry timings of retry_timeout: 100s.

N-th retry
Elapsed

1th

1s

2th

3s

3th

7s

4th

15s

5th

31s

6th

63s

7th

100s

retry_type

Specifies how to wait for the next retry to flush buffer.

Supported types:

  • exponential_backoff (default)

  • periodic

retry_forever

If true, plugin will ignore retry_timeout and retry_max_times options and retry flushing forever.

Default: false

retry_timeout

The maximum seconds to retry to flush while failing, until the plugin discards the buffer chunks. If the next retry is going to exceed this time limit, the last retry will be made at exactly this time limit.

Default: 72h (72 hours)

retry_max_times

The maximum number of times to retry to flush while failing. If retry_timeout is the default, the number is 18 with exponential backoff.

Default: nil

retry_secondary_threshold

The ratio of retry_timeout to switch to use secondary while failing.

Default: 0.8

retry_wait

Seconds to wait before the next retry to flush, or constant factor of exponential backoff.

Default: 1

retry_exponential_backoff_base

The base number of exponential backoff for retries.

Default: 2

retry_max_interval

The maximum interval (seconds) for exponential backoff between retries while failing.

Default: nil

retry_randomize

If true, the output plugin will retry after a randomized interval not to do burst retries.

Default: true

Secondary Output

In buffered mode, the user can specify <secondary> with any output plugin in <match> configuration. If plugins continue to fail writing buffer chunks and exceeds the timeout threshold for retries, then output plugins will delegate the writing of the buffer chunk to the secondary plugin.

<secondary> is useful for backup when destination servers are unavailable, e.g. forward, mongo, etc. We strongly recommend out_secondary_file plugin for <secondary>.

This example sends logs to Elasticsearch using a file buffer /var/log/td-agent/buffer/elasticsearch, and any failure will be sent to /var/log/td-agent/error/ using my.logs for file names:

<match my.logs>
  @type elasticsearch
  host localhost
  port 9200
  logstash_format true
  <buffer>
    @type file
    path /var/log/td-agent/buffer/elasticsearch
  </buffer>
  <secondary>
    @type secondary_file
    directory /var/log/td-agent/error
    basename my.logs
  </secondary>
</match>

NOTE: <secondary> plugin receives the primary's buffer chunk directly. So, you need to check if your secondary plugin works with the primary setting.

The retry timings of retry_timeout: 100s with the secondary.

N-th retry
Elapsed
Output plugin

1th

1s

primary

2th

3s

primary

3th

7s

primary

4th

15s

primary

5th

31s

primary

6th

63s

primary

7th

80s

secondary

8th

81s

secondary

9th

83s

secondary

10th

87s

secondary

11th

95s

secondary

12th

100s

secondary

The retry timings of retry_max_times: 10 with the secondary.

N-th retry
Elapsed
Output plugin

1th

1s

primary

2th

3s

primary

3th

7s

primary

4th

15s

primary

5th

31s

primary

6th

63s

primary

7th

127s

primary

8th

255s

primary

9th

511s

primary

10th

818s

secondary

Output plugins in v1 can control keys of buffer chunking by configurations, dynamically. Users can configure buffer chunk keys as time (any unit specified by user), tag and any key name of records. Output plugin will split events into chunks: events in a chunk have the same values for chunk keys. The output plugin's buffer behavior (if any) is defined by a separate . Different buffer plugins can be chosen for each output plugin.

/

See buffer section in for parameter name changes between v1 and v0.12.

See .

See for the behavior of the buffer.

For other configuration parameters available in <buffer> section, see .

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

Input
Parser
Filter
Output
Formatter
Storage
Service Discovery
Buffer
Metrics
Buffer plugin
out_copy
out_null
out_roundrobin
out_stdout
out_exec_filter
out_forward
out_mongo
out_mongo_replset
out_exec
out_file
out_s3
out_webhdfs
other plugins
Compat Parameters Plugin Helper API
Buffer Section Configurations
Buffer Plugin Overview
Buffer Plugin Overview
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
Fluentd v1.0 Plugin API Overview