out_kinesis
) to aggregate semi-structured logs in real-time. Kinesis plugin is developed and published by Amazon Web Services officially.ip
, path
, etc.) and buffers them.td-agent
package, please install it manually:/etc/td-agent/td-agent.conf
. Otherwise, it is located at /etc/fluentd/fluentd.conf
./var/log/apache2/access_log
). The Fluentd configuration file should look like this:in_tail
article for more information.@type tail
: The tail Input plugin continuously tracks the log@type apache2
in <parse>
: Uses Fluentd's built-in Apache log parser.path /var/log/apache2/access_log
: The location of the Apache log.tag kinesis.apache.access
: kinesis.apache.access
is used as the<match>...</match>
is used (i.e. the log is routed according to the config inside). In this example, the kinesis.apache.access
tag (generated by tail
) is always used.**
in match.**
matches zero or more period-delimited tag parts (e.g. match
/match.a
/match.a.b
).flush_interval
parameter specifies how often the data is written to Kinesis.random_partition_key true
option will generate the partition key via UUID v3 (source). Kinesis Stream consists of shards
, and the processing power of each shard is limited. This partition key will be used by Kinesis to determine which shard has been designated for a specific record.td-agent
to make sure that the configuration change is available:ab
(Apache Bench) program: