Fluentd
Fluentd
Introduction
Overview
Use Cases
Configuration
Deployment
Container Deployment
Input Plugins
Output Plugins
Buffer Plugins
Filter Plugins
Parser Plugins
Formatter Plugins
Developer
Articles
Store Apache Logs into MongoDB
Apache To Riak
Store Apache Logs into Amazon S3
Before Install
Cep Norikra
Collect Glusterfs Logs
Common Log Formats
Docker Logging Efk Compose
Docker Logging
Filter Modify Apache
Forwarding Over Ssl
Free Alternative To Splunk By Fluentd
Data Collection to Hadoop (HDFS)
Data Analytics with Treasure Data
Install By Chef
Install By Deb
Install By Dmg
Install By Gem
Install By Rpm
Install From Source
Install On Beanstalk
Install On Heroku
Java
Kinesis Stream
Kubernetes Fluentd
Monitoring Prometheus
Monitoring Rest Api
Nodejs
Performance Tuning Multi Process
Performance Tuning Single Process
Perl
Php
Python
Quickstart
Raspberrypi Cloud Data Logger
Recipe Apache Logs To Elasticsearch
Recipe Apache Logs To Mongo
Recipe Apache Logs To S3
Recipe Apache Logs To Treasure Data
Recipe Cloudstack To Mongodb
Recipe Csv To Elasticsearch
Recipe Csv To Mongo
Recipe Csv To S3
Recipe Csv To Treasure Data
Recipe Http Rest Api To Elasticsearch
Recipe Http Rest Api To Mongo
Recipe Http Rest Api To S3
Recipe Http Rest Api To Treasure Data
Recipe Json To Elasticsearch
Recipe Json To Mongo
Recipe Json To S3
Recipe Json To Treasure Data
Recipe Nginx To Elasticsearch
Recipe Nginx To Mongo
Recipe Nginx To S3
Recipe Nginx To Treasure Data
Recipe Syslog To Elasticsearch
Recipe Syslog To Mongo
Recipe Syslog To S3
Recipe Syslog To Treasure Data
Recipe Tsv To Elasticsearch
Recipe Tsv To Mongo
Recipe Tsv To S3
Recipe Tsv To Treasure Data
Ruby
Scala
Splunk Like Grep And Alert Email
Powered by GitBook

Recipe Csv To Treasure Data

Looking to get data out of csv into treasure data? You can do that with fluentd in 10 minutes!

Here is how:

$ gem install fluentd
$ gem install fluent-plugin-td
$ touch fluentd.conf

fluentd.conf should look like this (just copy and paste this into fluentd.conf):

<source>
  @type tail
  path /var/log/httpd-access.log #...or where you placed your Apache access log
  pos_file /var/log/td-agent/httpd-access.log.pos # This is where you record file position
  tag foobar.csv #fluentd tag!
  format csv
  keys key1, key2, key3 # e.g., user_id, timestamp, action
  time_key key2 # Specify the column that you want to use as timestamp
</source>
​
<match **>
  @type tdlog
  apikey <Treasure Data API key> # You get your API key by signing up for Treasure Data
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
</match>

After that, you can start fluentd and everything should work:

$ fluentd -c fluentd.conf

Of course, this is just a quick example. If you are thinking of running fluentd in production, consider using td-agent, the enterprise version of Fluentd packaged and maintained by Treasure Data, Inc..

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.

Previous
Recipe Csv To S3
Next
Recipe Http Rest Api To Elasticsearch
Last updated 11 months ago