secondary_file

The out_secondary_file Output plugin writes chunks to files. This plugin is similar to out_file but this is for <secondary> use-case.

NOTE: Do not use this plugin for the primary plugin.

out_secondary_file is included in Fluentd's core.

Example Configuration

<match pattern>
  @type forward
  # ...
  <secondary>
    @type secondary_file
    directory /var/log/fluentd/error
  </secondary>
</match>

With this configuration, failed buffer chunks are saved into /var/log/fluentd/error/dump.bin.N, N is 0-origin incremental number.

Please see the Configuration File article for the basic structure and syntax of the configuration file.

Plugin Helpers

No helpers.

Parameters

Common Parameters

@type (required)

The value must be secondary_file.

directory

type	default	version
string	required parameter	1.0.0

The directory path of the output file. Received buffer chunks are saved in this directory.

<secondary>
  @type secondary_file
  directory /var/log/fluentd/error
</secondary>

basename

type	default	version
string	dump.bin	1.0.0

The basename of the output file. You can use ${chunk_id} placeholder to identify the original chunk.

<secondary>
  @type secondary_file
  directory /var/log/fluentd/error
  basename dump.${chunk_id}
</secondary>

The output path would be:

# 59c278456e74a22dc594b06a7d4247c4 is chunk id
/var/log/fluentd/error/dump.59c278456e74a22dc594b06a7d4247c4.0

append

type	default	version
bool	false	1.0.0

Determines that the received chunk is appended to an existing file or not. By default, it is not appended and each chunk is flushed to a different path:

# append false
dump.bin.0
dump.bin.1
dump.bin.2
...
dump.bin.N

This makes parallel file processing easy. But if you want to disable this behavior, you can disable it by setting append true:

# append true
dump.bin

compress

type	default	version
enum	text	1.0.0

Supported compress types: text, gzip

When gzip is specified, .gz is automatically added to the output file path as its suffix.

How to resend secondary file

The secondary file can be resend by fluent-cat command.

Here is the example to resend dump.bin.0 to 127.0.0.1:24224 using fluent-cat.

$ cat /path/to/dump.bin.0 | fluent-cat --format msgpack  --host 127.0.0.1 --port 24224 TAG

See fluent-cat command line option about details.

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.