out_webhdfs
Output plugin writes records into HDFS (Hadoop Distributed File System). By default, it creates files on an hourly basis. This means that when you first import records using the plugin, no file is created immediately.timekey
condition has been met. To change the output frequency, please modify the timekey
value.out_webhdfs
is included in td-agent
by default (v1.1.10 or later). Fluentd gem users will have to install the fluent-plugin-webhdfs
gem using the following command:hdfs-site.xml
file and restart the whole cluster:<buffer>
, please check Buffer Section Configuration.@type
(required)webhfds
.host
(required)port
(required)path
(required)"#{Socket.gethostname}"
in your path to avoid writing into the same HDFS file from multiple Fluentd instances. This conflict could result in data loss.%Y
: year including the century (at least 4 digits)%m
: month of the year (01..12)%d
: Day of the month (01..31)%H
: Hour of the day, 24-hour clock (00..23)%M
: Minute of the hour (00..59)%S
: Second of the minute (00..60)path
configuration, it is recommended to specify the format using <format>
section.out_webhdfs
)timekey
timekey
parameter. The default value is 86400
(when the path does not contain the time placeholders), which creates one file per day.path
configuration. For example, when path
contains %H
, the value is 3600
and creates one file per hour.timekey_wait
10m
), where Fluentd will wait until 10 minutes past the hour for any logs that occurred within the past hour.queue_limit_length
, chunk_limit_size
64
and 8m
, respectively. The suffixes k
(KB), m
(MB), and g
(GB) can be used for chunk_limit_size
.flush_interval
s
(seconds), m
(minutes), and h
(hours) can be used.flush_at_shutdown
true
. Specify true
if you use memory
buffer type.retry_wait
, retry_max_interval
1.0
and unset
(no limit). The interval doubles (with +/-12.5% randomness) every retry until retry_max_interval
is reached.td-agent
will retry 17
times before giving up by default (see the retry_max_times
parameter for details), the sleep interval can be up to approximately 131072 seconds (roughly 36 hours) in the default configurations.retry_max_times
, retry_forever
true
, the value of retry_max_times
is ignored and there is no limit). The default values are 17
and false
(not disabled). If the limit is reached, buffered data is discarded and the retry interval is reset to its initial value (retry_wait
).flush_thread_count
1
.@log_level
@log_level
option allows the user to set different levels of logging for each plugin.fatal
, error
, warn
, info
, debug
, trace