file

The file buffer plugin provides a persistent buffer implementation. It uses files to store buffer chunks on disk.

Example Config

<match pattern>
  buffer_type file
  buffer_path /var/log/fluent/myapp.*.buffer
</match>

Please see the Config File article for the basic structure and syntax of the configuration file.

Parameters

buffer_type (required)

The value must be file.

buffer_path (required)

The path where buffer chunks are stored. The '*' is replaced with random characters. This parameter is require

This parameter must be unique to avoid race condition problem. For example, you can't use fixed buffer_path parameter in fluent-plugin-forest. ${tag} or similar placeholder is needed. Of course, this parameter must also be unique between fluentd instances.

In addition, buffer_path should not be an other buffer_path prefix. For example, the following conf doesn't work well. /var/log/fluent/foo resumes /var/log/fluent/foo.bar's buffer files during start phase and it causes No such file or directory in /var/log/fluent/foo.bar side.

<match pattern1>
    buffer_path /var/log/fluent/foo
</match>

<match pattern2>
    buffer_path /var/log/fluent/foo.bar
</match>

Here is the correct version to avoid prefix problem.

<match pattern1>
    buffer_path /var/log/fluent/foo.baz
</match>

<match pattern2>
    buffer_path /var/log/fluent/foo.bar
</match>

buffer_chunk_limit

The size of each buffer chunk. The default is 8m. The suffixes "k" (KB), "m" (MB), and "g" (GB) can be used. Please see the Buffer Plugin Overview article for the basic buffer structure.

The default value for Time Sliced Plugin is overwritten as 256m.

buffer_queue_limit

The length limit of the chunk queue. Please see the Buffer Plugin Overview article for the basic buffer structure. The default limit is 256 chunks.

flush_interval

The interval between data flushes. The suffixes "s" (seconds), "m" (minutes), and "h" (hours) can be used

flush_at_shutdown

If true, queued chunks are flushed at shutdown process. The default is false.

retry_wait

The interval between retries. The suffixes "s" (seconds), "m" (minutes), and "h" (hours) can be used.

Limitation

Caution:

file buffer implementation depends on the characteristics of local file system. Don't use file buffer on remote file system, e.g. NFS, GlusterFS, HDFS and etc. We observed major data loss by using remote file system.

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.