Versions | v0.12 (td-agent2) | v0.10 (td-agent1)

Getting Data From Syslog Into S3 Using Fluentd

Looking to get data out of syslog into s3? You can do that with fluentd in 10 minutes!

Here is how:

$ gem install fluentd
$ gem install fluent-plugin-s3
$ touch fluentd.conf

fluentd.conf should look like this (just copy and paste this into fluentd.conf):

  @type syslog
  port 5140
  tag system.local

<match **>
  @type s3
  path <s3 path> #(optional; default="")
  time_format <format string> #(optional; default is ISO-8601)
  aws_key_id <Your AWS key id> #(required)
  aws_sec_key <Your AWS secret key> #(required)
  s3_bucket <s3 bucket name> #(required)
  s3_endpoint <s3 endpoint name> #(required; ex:
  s3_object_key_format <format string> #(optional; default="%{path}%{time_slice}_%{index}.%{file_extension}")
  auto_create_bucket <true/false> #(optional; default=true)
  check_apikey_on_start <true/false> #(optional; default=true)
  proxy_uri <proxy uri string> #(optional)

After that, you can start fluentd and everything should work:

$ fluentd -c fluentd.conf

Of course, this is just a quick example. If you are thinking of running fluentd in production, consider using td-agent, the enterprise version of Fluentd packaged and maintained by Treasure Data, Inc..

Last updated: 2016-06-13 06:11:23 UTC

Versions | v0.12 (td-agent2) | v0.10 (td-agent1)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF), originally invented by Treasure Data, Inc. All components are available under the Apache 2 License.

Interested in the Fluentd Newsletters?