Versions | v1.0 (td-agent3) | v0.12 (td-agent2)

forward Output Plugin

The out_forward Buffered Output plugin forwards events to other fluentd nodes. This plugin supports load-balancing and automatic fail-over (a.k.a. active-active backup). For replication, please use the out_copy plugin.

The out_forward plugin detects server faults using a “φ accrual failure detector” algorithm. You can customize the parameters of the algorithm. When a server fault recovers, the plugin makes the server available automatically after a few seconds.

The out_forward plugin supports at-most-once and at-least-once semantics. The default is at-most-once.

Table of Contents

Example Configuration

out_forward is included in Fluentd’s core. No additional installation process is required.

<match pattern>
  @type forward
  send_timeout 60s
  recover_wait 10s
  hard_timeout 60s

  <server>
    name myserver1
    host 192.168.1.3
    port 24224
    weight 60
  </server>
  <server>
    name myserver2
    host 192.168.1.4
    port 24224
    weight 60
  </server>
  ...

  <secondary>
    @type file
    path /var/log/fluent/forward-failed
  </secondary>
</match>
Please see the Config File article for the basic structure and syntax of the configuration file.

Supported modes

  • Synchronous
  • Asynchronous

See Output Plugin Overview for more details.

Plugin helpers

Parameters

Common Parameters

@type

The value must be forward.

<server> (at least one is required)

required multi version
true true 0.14.5

The destination servers. Each server must have following information.

host

type default version
string required parameter 0.14.5

The IP address or host name of the server.

name

type default version
string nil 0.14.5

The name of the server. Used for logging and certificate verification in TLS transport (when host is address).

port

type default version
integer 24224 0.14.5

The port number of the host. Note that both TCP packets (event stream) and UDP packets (heartbeat message) are sent to this port.

shared_key

type default version
string nil 0.14.5

The shared key per server.

username

type default version
string “” (empty string) 0.14.5

The username for authentication.

password

type default version
string “” (empty string) 0.14.5

The password for authentication.

standby

type default version
bool false 0.14.5

Marks a node as the standby node for an Active-Standby model between Fluentd nodes. When an active node goes down, the standby node is promoted to an active node. The standby node is not used by the out_forward plugin until then.

<match pattern>
  @type forward
  ...

  <server>
    name myserver1
    host 192.168.1.3
    weight 60
  </server>
  <server>  # forward doesn't use myserver2 until myserver1 goes down
    name myserver2
    host 192.168.1.4
    weight 60
    standby
  </server>
  ...
</match>

weight

type default version
integer 60 0.14.5

The load balancing weight. If the weight of one server is 20 and the weight of the other server is 30, events are sent in a 2:3 ratio. The default weight is 60.

require_ack_response

type default version
bool false 0.14.0

Change the protocol to at-least-once. The plugin waits the ack from destination’s in_forward plugin.

ack_response_timeout

type default version
time 190 0.14.0

This option is used when require_ack_response is true. This default value is based on popular tcp_syn_retries.

If set 0, this plugin doesn’t wait the ack response.

send_timeout

type default version
time 60 0.14.0

The timeout time when sending event logs.

recover_wait

type default version
time 10 0.14.0

The wait time before accepting a server fault recovery.

heartbeat_type

type default available version
enum transport transport, tcp, udp, none 0.14.12

The transport protocol to use for heartbeats. Set “none” to disable heartbeat.

heartbeat_interval

type default version
time 1 0.14.0

The interval of the heartbeat packer.

phi_failure_detector

type default version
bool true 0.14.0

Use the “Phi accrual failure detector” to detect server failure.

phi_threshold

type default version
integer 16 0.14.0

The threshold parameter used to detect server faults.

`phi_threshold` is deeply related to `heartbeat_interval`. If you are using longer `heartbeat_interval`, please use the larger `phi_threshold`. Otherwise you will see frequent detachments of destination servers. The default value 16 is tuned for `heartbeat_interval` 1s.

hard_timeout

type default version
time 60 0.14.0

The hard timeout used to detect server failure. The default value is equal to the send_timeout parameter.

expire_dns_cache

type default version
time nil (persistent cache) 0.14.0

Set TTL to expire DNS cache in seconds. Set 0 not to use DNS Cache.

dns_round_robin

type default version
bool false 0.14.0

Enable client-side DNS round robin. Uniform randomly pick an IP address to send data when a hostname has several IP addresses.

`heartbeat_type udp` is not available with `dns_round_robin true`. Use `heartbeat_type tcp` or `heartbeat_type none`.

ignore_network_errors_at_startup

type default version
bool false 0.14.12

Ignore DNS resolution and errors at startup time.

tls_version

type default available version
enum TLSv1_2 TLSv1_1, TLSv1_2 0.14.12

The default version of TLS transport.

tls_ciphers

type default version
string ALL:!aNULL:!eNULL:!SSLv2 (OpenSSL > 1.0.0 default) 0.14.12

The cipher configuration of TLS transport.

tls_insecure_mode

type default version
bool false 0.14.12

Skip all verification of certificates or not.

tls_allow_self_signed_cert

type default version
bool false 0.14.12

Allow self signed certificates or not.

tls_verify_hostname

type default version
bool true 0.14.12

Verify hostname of servers and certificates or not in TLS transport.

tls_cert_path

type default version
array of string false 0.14.12

The additional CA certificate path for TLS.

<security> section

required multi version
false false 0.14.5

This section contains parameters related to authentication.

self_hostname

type default version
string required parameter 0.14.5

The hostname.

shared_key

type default version
string required parameter 0.14.5

Shared key for authentication.

<secondary>

required multi version
false false 0.14.0

The backup destination that is used when all servers are unavailable.

For more details, see Secondary Output.

Tips & Tricks

How to connect to a TLS/SSL enabled server

If you’ve set up TLS/SSL encryption in the receiving server, you need to tell the output forwarder to use encryption by setting the transport parameter:

<match debug.**>
  @type forward
  transport tls
  <server>
    host 192.168.1.2
    port 24224
  </server>
</match>

If you’re using a self-singed certificate, copy the certificate file to the forwarding server, then add the following settings:

<match debug.**>
  @type forward
  transport tls
  tls_cert_path /path/to/fluentd.crt # Set the path to the certificate file.
  tls_verify_hostname true           # Set false to ignore cert hostname.
  <server>
    host 192.168.1.2
    port 24224
  </server>
</match>

After updating the settings, please confirm that the forwarded data is being received by the destination node properly.

How to Enable Password Authentication

If you want to connect to a server that requires password authentication, you need to set your credentials in the configuration file.

<match debug.**>
  @type forward
  <server>
    host 192.168.1.2
    port 24224
  </server>
  <security>
    self_hostname HOSTNAME
    shared_key secret
  </security>
</match>

Note that, as to the option self_hostname, you need to set the name of the server on which your out_forward instance is running. In the current implementation, it is considered invalid if your in_forward and out_forward shares the same hostname.

How to enable gzip compression

Since v0.14.7, Fluentd supports transparent data compression. You can use this feature to reduce the transferred payload size.

To enable this feature, set the compress option as follows:

<match debug.**>
  @type forward
  compress gzip
  <server>
    host 192.168.1.2
    port 24224
  </server>
</match>

You don’t need any configuration in the receiving server. Data compression is auto-detected and handled transparently by the destination node.

What is a Phi accrual failure detector?

Fluentd implements an adaptive failure detection mechanism called “Phi accrual failure detector”. Here is how it works:

  1. Each in_forward node sends heartbeat packets to its out_foward server at a regular interval.
  2. The out_forward server records the arrival time of heartbeat packets sent by each node.
  3. If the server does not receive a heartbeat from one of its nodes for “a long time”, it assumes the node is down.

But how long should the server wait before detaching a node? The phi accrual failure detector answers this question by computing the probability of a node being down based on the assumption that heartbeat intervals follow normal distribution. Internally it represent the confidence of a node being down by a continuous function φ(t) which grows as the time from the last packet increases.

For example, suppose that the historical average interval is 1 seconds and the standard deviation is 1, it’s not likely that the node is still being active when its heartbeat does not arrive for the last 10 seconds.

For details, please read the original paper: Hayashibara, Naohiro, et al. “The φ accrual failure detector.” IEEE, 2004.

Troubleshooting

“no nodes are available”

Please make sure that you can communicate with port 24224 using not only TCP, but also UDP. These commands will be useful for checking the network configuration.

$ telnet host 24224
$ nmap -p 24224 -sU host

Please note that there is one known issue where VMware will occasionally lose small UDP packets used for heartbeat.

Last updated: 2018-11-18 18:22:36 +0000

Versions | v1.0 (td-agent3) | v0.12 (td-agent2)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.