Fluentd
Search…
0.12
Powered By GitBook
Recipe Json To S3
Looking to get data out of json into s3? You can do that with fluentd in 10 minutes!
Here is how:
1
$ gem install fluentd
2
$ gem install fluent-plugin-s3
3
$ touch fluentd.conf
Copied!
fluentd.conf should look like this (just copy and paste this into fluentd.conf):
1
<source>
2
@type tail
3
path /var/log/httpd-access.log #...or where you placed your Apache access log
4
pos_file /var/log/td-agent/httpd-access.log.pos # This is where you record file position
5
tag foobar.json #fluentd tag!
6
format json # one JSON per line
7
time_key time_field # optional; default = time
8
</source>
9
10
<match **>
11
@type s3
12
path <s3 path> #(optional; default="")
13
time_format <format string> #(optional; default is ISO-8601)
14
aws_key_id <Your AWS key id> #(required)
15
aws_sec_key <Your AWS secret key> #(required)
16
s3_bucket <s3 bucket name> #(required)
17
s3_endpoint <s3 endpoint name> #(required; ex: s3-us-west-1.amazonaws.com)
18
s3_object_key_format <format string> #(optional; default="%{path}%{time_slice}_%{index}.%{file_extension}")
19
auto_create_bucket <true/false> #(optional; default=true)
20
check_apikey_on_start <true/false> #(optional; default=true)
21
proxy_uri <proxy uri string> #(optional)
22
</match>
Copied!
After that, you can start fluentd and everything should work:
1
$ fluentd -c fluentd.conf
Copied!
Of course, this is just a quick example. If you are thinking of running fluentd in production, consider using td-agent, the enterprise version of Fluentd packaged and maintained by Treasure Data, Inc..
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 2yr ago
Copy link