Recipe Apache Logs To S3

Looking to get data out of apache logs into s3? You can do that with fluentd in 10 minutes!
Here is how:
$ gem install fluentd
$ gem install fluent-plugin-s3
$ touch fluentd.conf
fluentd.conf should look like this (just copy and paste this into fluentd.conf):
@type tail
path /var/log/httpd-access.log #...or where you placed your Apache access log
pos_file /var/log/td-agent/httpd-access.log.pos # This is where you record file position
tag apache.access #fluentd tag!
format apache2 # Do you have a custom format? You can write your own regex.
<match **>
@type s3
path <s3 path> #(optional; default="")
time_format <format string> #(optional; default is ISO-8601)
aws_key_id <Your AWS key id> #(required)
aws_sec_key <Your AWS secret key> #(required)
s3_bucket <s3 bucket name> #(required)
s3_endpoint <s3 endpoint name> #(required; ex: s3-us-west-1.amazonaws.com)
s3_object_key_format <format string> #(optional; default="%{path}%{time_slice}_%{index}.%{file_extension}")
auto_create_bucket <true/false> #(optional; default=true)
check_apikey_on_start <true/false> #(optional; default=true)
proxy_uri <proxy uri string> #(optional)
After that, you can start fluentd and everything should work:
$ fluentd -c fluentd.conf
Of course, this is just a quick example. If you are thinking of running fluentd in production, consider using td-agent, the enterprise version of Fluentd packaged and maintained by Treasure Data, Inc..
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.