forward
The out_forward
Buffered Output plugin forwards events to other fluentd nodes. This plugin supports load-balancing and automatic fail-over (i.e. active-active backup). For replication, please use out_copy
plugin.
The out_forward
plugin detects server faults using a φ accrual failure detector algorithm. You can customize the parameters of the algorithm. When a server fault recovers, the plugin makes the server available automatically after a few seconds.
The out_forward
plugin supports at-most-once and at-least-once semantics. The default is at-most-once.
It is included in Fluentd's core.
Example Configuration
Please see the Configuration File article for the basic structure and syntax of the configuration file.
Supported Modes
Synchronous
Asynchronous
See Output Plugin Overview for more details.
Plugin Helpers
Parameters
@type
@type
The value must be forward
.
<server>
(at least one is required)
<server>
(at least one is required)required | multi | version |
---|---|---|
true | true | 0.14.5 |
The destination servers. Each server has the following parameters:
host
name
port
shared_key
username
password
standby
weight
If you want to manage destination servers by flexible approach, use <service_discovery>
instead.
host
host
type | default | version |
---|---|---|
string | required parameter | 0.14.5 |
The IP address or host name of the server.
name
name
type | default | version |
---|---|---|
string | nil | 0.14.5 |
The name of the server. Used for logging and certificate verification in TLS transport (when the host is the address).
port
port
type | default | version |
---|---|---|
integer | 24224 | 0.14.5 |
The port number of the host. Note that both TCP packets (event stream) and UDP packets (heartbeat messages) are sent to this port.
shared_key
shared_key
type | default | version |
---|---|---|
string | nil | 0.14.5 |
The shared key per server.
username
username
type | default | version |
---|---|---|
string | "" (empty string) | 0.14.5 |
The username for authentication.
password
password
type | default | version |
---|---|---|
string | "" (empty string) | 0.14.5 |
The password for authentication.
standby
standby
type | default | version |
---|---|---|
bool | false | 0.14.5 |
Marks a node as the standby node for an Active-Standby model between Fluentd nodes. When an active node goes down, the standby node is promoted to an active node. The standby node is not used by the out_forward
plugin until then.
weight
weight
type | default | version |
---|---|---|
integer | 60 | 0.14.5 |
The load balancing weight. If the weight of one server is 20 and the weight of the other server is 30, events are sent in a 2:3 ratio. The default weight is 60.
<service_discovery>
<service_discovery>
Use service discovery plugin instead of fixed <server>
list. See also Service Discovery Plugin Overview for more details.
require_ack_response
require_ack_response
type | default | version |
---|---|---|
bool | false | 0.14.0 |
Changes the protocol to at-least-once. The plugin waits the ack from destination's in_forward
plugin.
ack_response_timeout
ack_response_timeout
type | default | version |
---|---|---|
time | 190 | 0.14.0 |
This option is used when require_ack_response
is true
. This default value is based on popular tcp_syn_retries
.
If set 0
, this plugin does not wait for the ack response.
send_timeout
send_timeout
type | default | version |
---|---|---|
time | 60 | 0.14.0 |
The timeout time when sending event logs.
connect_timeout
connect_timeout
type | default | version |
---|---|---|
time | nil(no timeout) | 1.6.0 |
The connection timeout for the socket. When the connection is timed out during the connection establishment, Errno::ETIMEDOUT
error is raised.
recover_wait
recover_wait
type | default | version |
---|---|---|
time | 10 | 0.14.0 |
The wait time before accepting a server fault recovery.
heartbeat_type
heartbeat_type
type | default | available | version |
---|---|---|---|
enum | transport | transport, tcp, udp, none | 0.14.12 |
Specifies the transport protocol for heartbeats. Set none
to disable.
heartbeat_interval
heartbeat_interval
type | default | version |
---|---|---|
time | 1 | 0.14.0 |
The interval of the heartbeat packer.
phi_failure_detector
phi_failure_detector
type | default | version |
---|---|---|
bool | true | 0.14.0 |
Use the "Phi accrual failure detector" to detect server failure.
phi_threshold
phi_threshold
type | default | version |
---|---|---|
integer | 16 | 0.14.0 |
The threshold parameter used to detect server faults.
phi_threshold
is directly related to heartbeat_interval
. If you are using longer heartbeat_interval
, please use the larger phi_threshold
. Otherwise, you will see frequent detachments of destination servers. The default value 16 is tuned for heartbeat_interval
1s.
hard_timeout
hard_timeout
type | default | version |
---|---|---|
time | 60 | 0.14.0 |
The hard timeout used to detect server failure. The default value is equal to the send_timeout
parameter.
expire_dns_cache
expire_dns_cache
type | default | version |
---|---|---|
time | nil (persistent cache) | 0.14.0 |
Sets TTL to expire DNS cache in seconds. Set 0 not to use DNS Cache.
dns_round_robin
dns_round_robin
type | default | version |
---|---|---|
bool | false | 0.14.0 |
Enable client-side DNS round robin. Uniform randomly pick an IP address to send data when a hostname has several IP addresses.
heartbeat_type udp
is not available with dns_round_robintrue
. Use heartbeat_type tcp
or heartbeat_type none
.
ignore_network_errors_at_startup
ignore_network_errors_at_startup
type | default | version |
---|---|---|
bool | false | 0.14.12 |
Ignores DNS resolution and errors at startup time.
tls_version
tls_version
type | default | available | version |
---|---|---|---|
enum |
|
| 0.14.12 |
The default version of TLS transport.
tls_ciphers
tls_ciphers
type | default | version |
---|---|---|
string | ALL:!aNULL:!eNULL:!SSLv2 (OpenSSL > 1.0.0 default) | 0.14.12 |
The cipher configuration of TLS transport.
tls_insecure_mode
tls_insecure_mode
type | default | version |
---|---|---|
bool | false | 0.14.12 |
Skips all verification of certificates or not.
tls_allow_self_signed_cert
tls_allow_self_signed_cert
type | default | version |
---|---|---|
bool | false | 0.14.12 |
Allows self-signed certificates or not.
tls_verify_hostname
tls_verify_hostname
type | default | version |
---|---|---|
bool | true | 0.14.12 |
Verifies hostname of servers and certificates or not in TLS transport.
If the following conditions are met, you must set tls_verify_hostname false
explicitly to forward events correctly:
specify
host
in<server>
section with IP address, not hostnamespecify server certificate file with
tls_cert_path
which contains common name (CN) field with IP address, not hostname
tls_cert_path
tls_cert_path
type | default | version |
---|---|---|
array of string | nil | 0.14.12 |
The additional CA certificate path for TLS.
tls_client_cert_path
tls_client_cert_path
type | default | version |
---|---|---|
string | nil | 1.3.2 |
The client certificate path for TLS.
tls_client_private_key_path
tls_client_private_key_path
type | default | version |
---|---|---|
string | nil | 1.3.2 |
The client private key path for TLS.
tls_client_private_key_passphrase
tls_client_private_key_passphrase
type | default | version |
---|---|---|
string | nil | 1.3.2 |
The TLS private key passphrase for the client.
tls_cert_thumbprint
tls_cert_thumbprint
type | default | version |
---|---|---|
string | nil | 1.7.1 |
The certificate thumbprint for searching from Windows system certstore. This parameter is for Windows only.
tls_cert_logical_store_name
tls_cert_logical_store_name
type | default | version |
---|---|---|
string | nil | 1.7.1 |
The certificate logical store name on Windows system certstore. This parameter is for Windows only.
tls_cert_use_enterprise_store
tls_cert_use_enterprise_store
type | default | version |
---|---|---|
string | true | 1.7.1 |
Enables the certificate enterprise store on Windows system certstore. This parameter is for Windows only.
keepalive
keepalive
type | default | version |
---|---|---|
bool | false | 1.5.0 |
Enables the keepalive connection.
keepalive_timeout
keepalive_timeout
type | default | version |
---|---|---|
time | nil | 1.5.0 |
Timeout for keepalive. Default value is nil
which means to keep the connection alive as long as possible.
<security>
Section
<security>
Sectionrequired | multi | version |
---|---|---|
false | false | 0.14.5 |
This section contains parameters related to authentication:
self_hostname
shared_key
self_hostname
self_hostname
type | default | version |
---|---|---|
string | required parameter | 0.14.5 |
The hostname.
shared_key
shared_key
type | default | version |
---|---|---|
string | required parameter | 0.14.5 |
The shared key for authentication. If you want to specify shared_key
for specific server, use the <server>
section.
<secondary>
<secondary>
required | multi | version |
---|---|---|
false | false | 0.14.0 |
The backup destination that is used when all servers are unavailable.
For more details, see Secondary Output.
verify_connection_at_startup
verify_connection_at_startup
type | default | version |
---|---|---|
bool | false | 1.3.1 |
Verify that a connection can be made with one of out_forward
nodes at the time of startup.
<buffer>
<buffer>
See Buffer Section Configurations for more details.
chunk_keys
chunk_keys
type | default | version |
---|---|---|
array | tag | 0.14.9 |
Overwrites the default value in this plugin.
Tips and Tricks
How to connect to a TLS/SSL enabled server?
If you have set up TLS/SSL encryption for the receiving server, you need to tell the output forwarder to use encryption by setting the transport
parameter:
If you are using a self-signed certificate, copy the certificate file to the forwarding server, then add the following settings:
After updating the settings, please confirm that the forwarded data is being received by the destination node properly.
How to connect to a TLS/SSL enabled server with Windows Certstore Certificate?
If you have set up TLS/SSL encryption in the receiving server, you need to tell the output forwarder to use encryption by setting the transport
parameter.
Valid logical store names are:
MY
CA
ROOT
AUTHROOT
DISALLOWED
SPC
TRUST
TRUSTEDPEOPLE
TRUSTEDPUBLISHER
CLIENTAUTHISSUER
TRUSTEDDEVICES
SMARTCARDROOT
WEBHOSTING
REMOTE DESKTOP
Logical store name is case-insensitive. Note that this section configurations work only for Windows.
If you are using a self-signed certificate, export the certificate file from Windows Certstore and copy to the forwarding server, then add the following settings:
Note that these configuration works for root certificate which is put in Windows Certstore. Currently, the chained certificate is not supported.
Certificate thumbprint is able to obtain with certutil
command.
Here is an example which uses a certificate that is generated by fluent-ca-generate
command:
Cert hash(sha1)
is called as thumbprint in this section.
Note that -enterprise
flag represents to use enterprise certstore. Please pay attention to using whether enterprise certificates store or not.
How to Enable Password Authentication?
If you want to connect to a server that requires password authentication, you need to set your credentials in the configuration file:
Note that, as to the option self_hostname
, you need to set the name of the server on which your out_forward
instance is running. In the current implementation, it is considered invalid if your in_forward
and out_forward
share the same hostname
.
How to enable gzip
compression?
gzip
compression?Since v0.14.7, Fluentd supports transparent data compression. You can use this feature to reduce the transferred payload size.
To enable this feature, set the compress
option as follows:
You do not need any configuration in the receiving server. Data compression is auto-detected and handled transparently by the destination node.
What is a Phi accrual failure detector?
Fluentd implements an adaptive failure detection mechanism called "Phi accrual failure detector". Here is how it works:
Each
in_forward
node sends heartbeat packets to itsout_forward
serverwith a regular interval.
The
out_forward
server records the arrival time of heartbeat packets sentby each node.
If the server does not receive a heartbeat from one of its nodes for "a long
time", it assumes the node is down.
But how long should the server wait before detaching a node? The phi accrual failure detector answers this question by computing the probability of a node being down based on the assumption that heartbeat intervals follow normal distribution. Internally, it represents the confidence of a node being down by a continuous function φ(t) which grows as the time from the last packet increases.
For example, suppose that the historical average interval is 1 second and the standard deviation is 1, it is unlikely that the node is still being active when its heartbeat has not been received for the last 10 seconds.
For details, please read the original paper: Hayashibara, Naohiro, et al. "The φ accrual failure detector." IEEE, 2004.
Troubleshooting
"no nodes are available"
Please make sure that you can communicate with port 24224 using not only TCP, but also UDP. These commands will be useful for checking the network configuration:
Please note that there is one known issue where VMware will occasionally lose small UDP packets used for heartbeat.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last updated