Versions | v0.12 (td-agent2)

Getting Data From Tsv Into Treasure Data Using Fluentd

Looking to get data out of tsv into treasure data? You can do that with fluentd in 10 minutes!

Here is how:

$ gem install fluentd
$ gem install fluent-plugin-td
$ touch fluentd.conf

fluentd.conf should look like this (just copy and paste this into fluentd.conf):

  @type tail
  path /var/log/httpd-access.log #...or where you placed your Apache access log
  pos_file /var/log/td-agent/httpd-access.log.pos # This is where you record file position
  tag foobar.tsv #fluentd tag!
  format tsv
  keys key1, key2, key3 # e.g., user_id, timestamp, action
  time_key key2 # Specify the column that you want to use as timestamp

<match **>
  @type tdlog
  apikey <Treasure Data API key> # You get your API key by signing up for Treasure Data
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td

After that, you can start fluentd and everything should work:

$ fluentd -c fluentd.conf

Of course, this is just a quick example. If you are thinking of running fluentd in production, consider using td-agent, the enterprise version of Fluentd packaged and maintained by Treasure Data, Inc..

Last updated: 2016-06-13 06:11:23 UTC

Versions | v0.12 (td-agent2)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.