Fluentd
Search…
0.12
Recipe Tsv To Treasure Data
Looking to get data out of tsv into treasure data? You can do that with fluentd in 10 minutes!
Here is how:
1
$ gem install fluentd
2
$ gem install fluent-plugin-td
3
$ touch fluentd.conf
Copied!
fluentd.conf should look like this (just copy and paste this into fluentd.conf):
1
<source>
2
@type tail
3
path /var/log/httpd-access.log #...or where you placed your Apache access log
4
pos_file /var/log/td-agent/httpd-access.log.pos # This is where you record file position
5
tag foobar.tsv #fluentd tag!
6
format tsv
7
keys key1, key2, key3 # e.g., user_id, timestamp, action
8
time_key key2 # Specify the column that you want to use as timestamp
9
</source>
10
11
<match **>
12
@type tdlog
13
apikey <Treasure Data API key> # You get your API key by signing up for Treasure Data
14
auto_create_table
15
buffer_type file
16
buffer_path /var/log/td-agent/buffer/td
17
</match>
Copied!
After that, you can start fluentd and everything should work:
1
$ fluentd -c fluentd.conf
Copied!
Of course, this is just a quick example. If you are thinking of running fluentd in production, consider using td-agent, the enterprise version of Fluentd packaged and maintained by Treasure Data, Inc..
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Copy link