Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Example Configuration
  • Supported Modes
  • Plugin Helpers
  • Parameters
  • @type
  • <server> (at least one is required)
  • <service_discovery>
  • require_ack_response
  • ack_response_timeout
  • send_timeout
  • connect_timeout
  • recover_wait
  • heartbeat_type
  • heartbeat_interval
  • phi_failure_detector
  • phi_threshold
  • hard_timeout
  • expire_dns_cache
  • dns_round_robin
  • ignore_network_errors_at_startup
  • tls_version
  • tls_ciphers
  • tls_insecure_mode
  • tls_allow_self_signed_cert
  • tls_verify_hostname
  • tls_cert_path
  • tls_client_cert_path
  • tls_client_private_key_path
  • tls_client_private_key_passphrase
  • tls_cert_thumbprint
  • tls_cert_logical_store_name
  • tls_cert_use_enterprise_store
  • keepalive
  • keepalive_timeout
  • <security> Section
  • <secondary>
  • verify_connection_at_startup
  • <buffer>
  • Tips and Tricks
  • How to connect to a TLS/SSL enabled server?
  • How to connect to a TLS/SSL enabled server with Windows Certstore Certificate?
  • How to Enable Password Authentication?
  • How to enable gzip compression?
  • What is a Phi accrual failure detector?
  • Troubleshooting
  • "no nodes are available"

Was this helpful?

  1. Output Plugins

forward

PreviousfileNexthttp

Last updated 1 year ago

Was this helpful?

The out_forward Buffered Output plugin forwards events to other fluentd nodes. This plugin supports load-balancing and automatic fail-over (i.e. active-active backup). For replication, please use plugin.

The out_forward plugin detects server faults using a φ accrual failure detector algorithm. You can customize the parameters of the algorithm. When a server fault recovers, the plugin makes the server available automatically after a few seconds.

The out_forward plugin supports at-most-once and at-least-once semantics. The default is at-most-once.

It is included in Fluentd's core.

Example Configuration

<match pattern>
  @type forward
  send_timeout 60s
  recover_wait 10s
  hard_timeout 60s

  <server>
    name myserver1
    host 192.168.1.3
    port 24224
    weight 60
  </server>
  <server>
    name myserver2
    host 192.168.1.4
    port 24224
    weight 60
  </server>
  ...

  <secondary>
    @type file
    path /var/log/fluent/forward-failed
  </secondary>
</match>

Supported Modes

  • Synchronous

  • Asynchronous

Plugin Helpers

Parameters

@type

The value must be forward.

<server> (at least one is required)

required
multi
version

true

true

0.14.5

The destination servers. Each server has the following parameters:

  • host

  • name

  • port

  • shared_key

  • username

  • password

  • standby

  • weight

If you want to manage destination servers by flexible approach, use <service_discovery> instead.

host

type
default
version

string

required parameter

0.14.5

The IP address or host name of the server.

name

type
default
version

string

nil

0.14.5

The name of the server. Used for logging and certificate verification in TLS transport (when the host is the address).

port

type
default
version

integer

24224

0.14.5

The port number of the host. Note that both TCP packets (event stream) and UDP packets (heartbeat messages) are sent to this port.

shared_key

type
default
version

string

nil

0.14.5

The shared key per server.

username

type
default
version

string

"" (empty string)

0.14.5

The username for authentication.

password

type
default
version

string

"" (empty string)

0.14.5

The password for authentication.

standby

type
default
version

bool

false

0.14.5

Marks a node as the standby node for an Active-Standby model between Fluentd nodes. When an active node goes down, the standby node is promoted to an active node. The standby node is not used by the out_forward plugin until then.

<match pattern>
  @type forward
  # ...

  <server>
    name myserver1
    host 192.168.1.3
    weight 60
  </server>
  <server>  # forward doesn't use myserver2 until myserver1 goes down
    name myserver2
    host 192.168.1.4
    weight 60
    standby
  </server>
  # ...
</match>

weight

type
default
version

integer

60

0.14.5

The load balancing weight. If the weight of one server is 20 and the weight of the other server is 30, events are sent in a 2:3 ratio. The default weight is 60.

<service_discovery>

<match pattern>
  @type forward

  <service_discovery>
    @type file
    path /path/to/servers.yaml
  </service_discovery>
</source>

require_ack_response

type
default
version

bool

false

0.14.0

Changes the protocol to at-least-once. The plugin waits the ack from destination's in_forward plugin.

ack_response_timeout

type
default
version

time

190

0.14.0

This option is used when require_ack_response is true. This default value is based on popular tcp_syn_retries.

If set 0, this plugin does not wait for the ack response.

send_timeout

type
default
version

time

60

0.14.0

The timeout time when sending event logs.

connect_timeout

type
default
version

time

nil(no timeout)

1.6.0

The connection timeout for the socket. When the connection is timed out during the connection establishment, Errno::ETIMEDOUT error is raised.

recover_wait

type
default
version

time

10

0.14.0

The wait time before accepting a server fault recovery.

heartbeat_type

type
default
available
version

enum

transport

transport, tcp, udp, none

0.14.12

Specifies the transport protocol for heartbeats. Set none to disable.

heartbeat_interval

type
default
version

time

1

0.14.0

The interval of the heartbeat packer.

phi_failure_detector

type
default
version

bool

true

0.14.0

Use the "Phi accrual failure detector" to detect server failure.

phi_threshold

type
default
version

integer

16

0.14.0

The threshold parameter used to detect server faults.

phi_threshold is directly related to heartbeat_interval. If you are using longer heartbeat_interval, please use the larger phi_threshold. Otherwise, you will see frequent detachments of destination servers. The default value 16 is tuned for heartbeat_interval 1s.

hard_timeout

type
default
version

time

60

0.14.0

The hard timeout used to detect server failure. The default value is equal to the send_timeout parameter.

expire_dns_cache

type
default
version

time

nil (persistent cache)

0.14.0

Sets TTL to expire DNS cache in seconds. Set 0 not to use DNS Cache.

dns_round_robin

type
default
version

bool

false

0.14.0

Enable client-side DNS round robin. Uniform randomly pick an IP address to send data when a hostname has several IP addresses.

heartbeat_type udp is not available with dns_round_robintrue. Use heartbeat_type tcp or heartbeat_type none.

ignore_network_errors_at_startup

type
default
version

bool

false

0.14.12

Ignores DNS resolution and errors at startup time.

tls_version

type
default
available
version

enum

TLSv1_2

TLSv1_1, TLSv1_2

0.14.12

The default version of TLS transport.

tls_ciphers

type
default
version

string

ALL:!aNULL:!eNULL:!SSLv2 (OpenSSL > 1.0.0 default)

0.14.12

The cipher configuration of TLS transport.

tls_insecure_mode

type
default
version

bool

false

0.14.12

Skips all verification of certificates or not.

tls_allow_self_signed_cert

type
default
version

bool

false

0.14.12

Allows self-signed certificates or not.

tls_verify_hostname

type
default
version

bool

true

0.14.12

Verifies hostname of servers and certificates or not in TLS transport.

If the following conditions are met, you must set tls_verify_hostname false explicitly to forward events correctly:

  • specify host in <server> section with IP address, not hostname

  • specify server certificate file with tls_cert_path which contains common name (CN) field with IP address, not hostname

tls_cert_path

type
default
version

array of string

nil

0.14.12

The additional CA certificate path for TLS.

tls_client_cert_path

type
default
version

string

nil

1.3.2

The client certificate path for TLS.

tls_client_private_key_path

type
default
version

string

nil

1.3.2

The client private key path for TLS.

tls_client_private_key_passphrase

type
default
version

string

nil

1.3.2

The TLS private key passphrase for the client.

tls_cert_thumbprint

type
default
version

string

nil

1.7.1

The certificate thumbprint for searching from Windows system certstore. This parameter is for Windows only.

tls_cert_logical_store_name

type
default
version

string

nil

1.7.1

The certificate logical store name on Windows system certstore. This parameter is for Windows only.

tls_cert_use_enterprise_store

type
default
version

string

true

1.7.1

Enables the certificate enterprise store on Windows system certstore. This parameter is for Windows only.

keepalive

type
default
version

bool

false

1.5.0

Enables the keepalive connection.

keepalive_timeout

type
default
version

time

nil

1.5.0

Timeout for keepalive. Default value is nil which means to keep the connection alive as long as possible.

<security> Section

required
multi
version

false

false

0.14.5

This section contains parameters related to authentication:

  • self_hostname

  • shared_key

self_hostname

type
default
version

string

required parameter

0.14.5

The hostname.

shared_key

type
default
version

string

required parameter

0.14.5

The shared key for authentication. If you want to specify shared_key for specific server, use the <server> section.

<secondary>

required
multi
version

false

false

0.14.0

The backup destination that is used when all servers are unavailable.

verify_connection_at_startup

type
default
version

bool

false

1.3.1

Verify that a connection can be made with one of out_forward nodes at the time of startup.

<buffer>

chunk_keys

type
default
version

array

tag

0.14.9

Overwrites the default value in this plugin.

Tips and Tricks

How to connect to a TLS/SSL enabled server?

<match debug.**>
  @type forward
  transport tls
  <server>
    host example.com
    port 24224
  </server>
</match>

If you are using a self-signed certificate, copy the certificate file to the forwarding server, then add the following settings:

<match debug.**>
  @type forward
  transport tls
  tls_cert_path /path/to/fluentd.crt # Set the path to the certificate file.
  <server>
    # Set the remote server name. This name should match the Common Name
    # field in the certificate.
    host example.com
    port 24224
  </server>
</match>

After updating the settings, please confirm that the forwarded data is being received by the destination node properly.

How to connect to a TLS/SSL enabled server with Windows Certstore Certificate?

Valid logical store names are:

  • MY

  • CA

  • ROOT

  • AUTHROOT

  • DISALLOWED

  • SPC

  • TRUST

  • TRUSTEDPEOPLE

  • TRUSTEDPUBLISHER

  • CLIENTAUTHISSUER

  • TRUSTEDDEVICES

  • SMARTCARDROOT

  • WEBHOSTING

  • REMOTE DESKTOP

Logical store name is case-insensitive. Note that this section configurations work only for Windows.

<match debug.**>
  @type forward
  transport tls
  # Set valid logical store name.
  tls_cert_logical_store_name Trust
  tls_verify_hostname true # Set false to ignore the cert hostname.
  <server>
    host 192.168.1.2
    port 24224
  </server>
</match>

If you are using a self-signed certificate, export the certificate file from Windows Certstore and copy to the forwarding server, then add the following settings:

<match debug.**>
  @type forward
  transport tls
  # Set certificate SHA1 hash which is usually called as thumbprint.
  tls_cert_thumbprint <YOUR CERTIFICATE THUMBPRINT>
  tls_cert_logical_store_name Trust
  tls_verify_hostname true # Set false to ignore cert hostname.
  tls_cert_use_enterprise_store true # Set false to use non-enterprise certificate.
  <server>
    host 192.168.1.2
    port 24224
  </server>
</match>

Note that these configuration works for root certificate which is put in Windows Certstore. Currently, the chained certificate is not supported.

Here is an example which uses a certificate that is generated by fluent-ca-generate command:

PS> certutil -store -enterprise <YOUR CERTIFICATE STORE NAME>
trust "Enterprise Trust"
================ Certificate 0 ================
Serial Number: 01
Issuer: CN=Fluentd Forward CA, L=Mountain View, S=CA, C=US
 NotBefore: 1/1/1970 12:00 AM
 NotAfter: 8/21/2024 7:43 AM
Subject: CN=Fluentd Forward CA, L=Mountain View, S=CA, C=US
Signature matches Public Key
Root Certificate: Subject matches Issuer
Cert Hash(sha1): <YOUR CERTIFICATE THUMBPRINT>
No key provider information
Cannot find the certificate and private key for decryption.
CertUtil: -store command completed successfully.

Cert hash(sha1) is called as thumbprint in this section.

Note that -enterprise flag represents to use enterprise certstore. Please pay attention to using whether enterprise certificates store or not.

How to Enable Password Authentication?

<match debug.**>
  @type forward
  <server>
    host 192.168.1.2
    port 24224
  </server>
  <security>
    self_hostname HOSTNAME
    shared_key secret
  </security>
</match>

Note that, as to the option self_hostname, you need to set the name of the server on which your out_forward instance is running. In the current implementation, it is considered invalid if your in_forward and out_forward share the same hostname.

How to enable gzip compression?

Since v0.14.7, Fluentd supports transparent data compression. You can use this feature to reduce the transferred payload size.

To enable this feature, set the compress option as follows:

<match debug.**>
  @type forward
  compress gzip
  <server>
    host 192.168.1.2
    port 24224
  </server>
</match>

You do not need any configuration in the receiving server. Data compression is auto-detected and handled transparently by the destination node.

What is a Phi accrual failure detector?

Fluentd implements an adaptive failure detection mechanism called "Phi accrual failure detector". Here is how it works:

  1. Each in_forward node sends heartbeat packets to its out_forward server

    with a regular interval.

  2. The out_forward server records the arrival time of heartbeat packets sent

    by each node.

  3. If the server does not receive a heartbeat from one of its nodes for "a long

    time", it assumes the node is down.

But how long should the server wait before detaching a node? The phi accrual failure detector answers this question by computing the probability of a node being down based on the assumption that heartbeat intervals follow normal distribution. Internally, it represents the confidence of a node being down by a continuous function φ(t) which grows as the time from the last packet increases.

For example, suppose that the historical average interval is 1 second and the standard deviation is 1, it is unlikely that the node is still being active when its heartbeat has not been received for the last 10 seconds.

Troubleshooting

"no nodes are available"

Please make sure that you can communicate with port 24224 using not only TCP, but also UDP. These commands will be useful for checking the network configuration:

$ telnet host 24224
$ nmap -p 24224 -sU host

Please see the article for the basic structure and syntax of the configuration file.

See for more details.

Use service discovery plugin instead of fixed <server> list. See also for more details.

For more details, see .

See for more details.

If you have set up for the receiving server, you need to tell the output forwarder to use encryption by setting the transport parameter:

If you have set up in the receiving server, you need to tell the output forwarder to use encryption by setting the transport parameter.

Certificate thumbprint is able to obtain with command.

If you want to connect to a server that requires , you need to set your credentials in the configuration file:

For details, please read the original paper:

Please note that there is one where VMware will occasionally lose small UDP packets used for heartbeat.

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

Configuration File
Output Plugin Overview
socket
server
timer
thread
compat_parameters
Common Parameters
Service Discovery Plugin Overview
Buffer Section Configurations
certutil
Hayashibara, Naohiro, et al. "The φ accrual failure detector." IEEE, 2004.
known issue
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
out_copy
Secondary Output
TLS/SSL encryption
TLS/SSL encryption
password authentication