Versions | v0.12 (td-agent2) | v0.10 (td-agent1)

forward Input Plugin

The in_forward Input plugin listens to a TCP socket to receive the event stream. It also listens to an UDP socket to receive heartbeat messages.

This plugin is mainly used to receive event logs from other Fluentd instances, the fluent-cat command, or client libraries. This is by far the most efficient way to retrieve the records.

Do NOT use this plugin for inter-DC or public internet data transfer without secure connections. To provide the reliable / low-latency transfer, we assume this plugin is only used within private networks.

If you open up the port of this plugin to the internet, the attacker can easily crash Fluentd by using the specific packet. If you require a secure connection between nodes, please consider using in_secure_forward.

Table of Contents

Example Configuration

in_forward is included in Fluentd’s core. No additional installation process is required.

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
Please see the Config File article for the basic structure and syntax of the configuration file.

Parameters

type (required)

The value must be forward.

port

The port to listen to. Default Value = 24224

bind

The bind address to listen to. Default Value = 0.0.0.0 (all addresses)

linger_timeout

The timeout time used to set linger option. The default is 0

chunk_size_limit

The size limit of the the received chunk. If the chunk size is larger than this value, then the received chunk is dropped. The default is nil (no limit).

chunk_size_warn_limit

The warning size limit of the received chunk. If the chunk size is larger than this value, a warning message will be sent. The default is nil (no warning).

skip_invalid_event (v0.12.20 or later)

Skip an event if incoming event is invalid. The default is false.

This option is useful at forwarder, not aggragator. v0.12.20 or later.

source_hostname_key (v0.12.28 or later)

The field name of the client’s hostname. If set the value, the client’s hostname will be set to its key. The default is nil (no adding hostname).

This iterates incoming events. So if you sends larger chunks to in_forward, it needs additional processing time.

log_level option

The log_level option allows the user to set different levels of logging for each plugin. The supported log levels are: fatal, error, warn, info, debug, and trace.

Please see the logging article for further details.

Protocol

This plugin accepts both JSON or MessagePack messages and automatically detects which is used. Internally, Fluent uses MessagePack as it is more efficient than JSON.

The time value is a platform specific integer and is based on the output of Ruby’s Time.now.to_i function. On Linux, BSD and MAC systems, this is the number of seconds since 1970.

Multiple messages may be sent in the same connection.

stream:
  message...

message:
  [tag, time, record]
  or
  [tag, [[time,record], [time,record], ...]]

example:
  ["myapp.access", 1308466941, {"a":1}]["myapp.messages", 1308466942, {"b":2}]
  ["myapp.access", [[1308466941, {"a":1}], [1308466942, {"b":2}]]]

FAQ

How to parse incoming events?

in_forward doesn’t provide parsing mechanizm unlike in_tail or in_tcp because in_forward is mainly for efficient log transfer. If you want to parse incoming event, use parser filter in your pipeline.
See Docker logging driver usecase: Docker Logging

I got MessagePack::UnknownExtTypeError error. Why?

This error happens when forwarder’s fluentd is v0.14 and receiver’s fluentd is 0.12. To avoid this problem, set time_as_integer to out_forward setting in v0.14 fluentd.

Last updated: 2016-11-10 02:29:44 UTC

Versions | v0.12 (td-agent2) | v0.10 (td-agent1)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF), originally invented by Treasure Data, Inc. All components are available under the Apache 2 License.

Interested in the Fluentd Newsletters?