file - Fluentd

Search…
file
The file buffer plugin provides a persistent buffer implementation. It uses files to store buffer chunks on disk.
Parameters
​Common Parameters​
​Buffer Section Configurations​
path
type
required
default
version
string
✔
​
0.9.0
The path where buffer chunks are stored. The '*' is replaced with random characters. This parameter is required.
This parameter must be unique to avoid the race condition problem. For example, you cannot use a fixed path parameter in fluent-plugin-forest. ${tag} or similar placeholder is needed. Of course, this parameter must also be unique between fluentd instances.
In addition, path should not be another path prefix. For example, the following configuration does not work well. /var/log/fluent/foo resumes /var/log/fluent/foo.bar's buffer files during start phase and it causes No such file or directory in /var/log/fluent/foo.bar side.
1
<match pattern1>
2
  <buffer>
3
    @type file
4
    path /var/log/fluent/foo
5
  </buffer>
6
</match>
7
​
8
<match pattern2>
9
  <buffer>
10
    @type file
11
    path /var/log/fluent/foo.bar
12
  </buffer>
13
</match>
Copied!
Here is the correct version to avoid the prefix problem:
1
<match pattern1>
2
  <buffer>
3
    @type file
4
    path /var/log/fluent/foo.baz
5
  </buffer>
6
</match>
7
​
8
<match pattern2>
9
  <buffer>
10
    @type file
11
    path /var/log/fluent/foo.bar
12
  </buffer>
13
</match>
Copied!
Please make sure that you have enough space in the path directory. Running out of disk space is a problem frequently reported by users.
path_suffix
type
default
version
string
.log
1.6.3
Changes the suffix of the buffer file.
1
# default 
2
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
3
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta
4
​
5
# with 'path_suffix .buf'
6
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf
7
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf.meta
Copied!
This parameter is useful when .log is not fit for your environment. See also this issue's comment.
Example
1
<match pattern>
2
  <buffer>
3
    @type file
4
    path /var/log/fluent/myapp.*.buffer
5
  </buffer>
6
</match>
Copied!
Tips
Multi-Process Environment
If you use this plugin under the multi-process environment, you need to use @id/root_dir parameters instead of fixed path parameter. See Multi Process Workers article.
Limitation
Caution: file buffer implementation depends on the characteristics of the local file system. Don't use file buffer on remote file systems e.g. NFS, GlusterFS, HDFS, etc. We observed major data loss by using the remote file system.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
memory
file_single
Last modified 4mo ago
Copy link
Contents
Multi-Process Environment
Limitation
Parameters

`path`

`path_suffix`

Example

Tips

Multi-Process Environment

Limitation
