Recipe Http Rest Api To S3

Looking to get data out of http rest api into s3? You can do that with fluentd in 10 minutes!
Here is how:
1
$ gem install fluentd
2
$ gem install fluent-plugin-s3
3
$ touch fluentd.conf
Copied!
fluentd.conf should look like this (just copy and paste this into fluentd.conf):
1
<source>
2
  @type http
3
  port 8888
4
  bind 0.0.0.0
5
  body_size_limit 32m
6
  keepalive_timeout 10s
7
  # tag is part of the URL, e.g.,
8
  # curl -X POST -d 'json={"action":"login","user":2}' http://localhost:8888/tag.here
9
</source>
10
​
11
<match **>
12
  @type s3
13
  path <s3 path> #(optional; default="")
14
  time_format <format string> #(optional; default is ISO-8601)
15
  aws_key_id <Your AWS key id> #(required)
16
  aws_sec_key <Your AWS secret key> #(required)
17
  s3_bucket <s3 bucket name> #(required)
18
  s3_endpoint <s3 endpoint name> #(required; ex: s3-us-west-1.amazonaws.com)
19
  s3_object_key_format <format string> #(optional; default="%{path}%{time_slice}_%{index}.%{file_extension}")
20
  auto_create_bucket <true/false> #(optional; default=true)
21
  check_apikey_on_start <true/false> #(optional; default=true)
22
  proxy_uri <proxy uri string> #(optional)
23
</match>
Copied!
After that, you can start fluentd and everything should work:
1
$ fluentd -c fluentd.conf
Copied!
Of course, this is just a quick example. If you are thinking of running fluentd in production, consider using td-agent, the enterprise version of Fluentd packaged and maintained by Treasure Data, Inc..
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Recipe Http Rest Api To Mongo
Recipe Http Rest Api To Treasure Data
Last modified 2yr ago
Copy link