Install by DEB Package (Debian/Ubuntu)

Search…
This article explains how to install stable versions of td-agent deb packages, the stable Fluentd distribution packages maintained by Treasure Data, Inc and Calyptia, Inc..
What is td-agent?
Fluentd is written in Ruby for flexibility, with performance-sensitive parts in C. However, some users may have difficulty installing and operating a Ruby daemon.
That is why Treasure Data, Inc provides the stable distribution of Fluentd, called td-agent. The differences between Fluentd and td-agent can be found here.
What is calyptia-fluentd?
Our Calyptia also knows that Fluentd is written in Ruby for flexibility, with performance-sensitive parts in C. However, some users may have difficulty installing and operating a Ruby daemon. And td-agent is still seated on Ruby 2.7 due to compatibility reasons and Ruby versioning policy, calyptia-fluentd uses Ruby 3 instead of Ruby 2.7 for now.
That is why Calyptia, Inc. provides the alternative stable distribution of Fluentd, called calyptia-fluentd. The differences between td-agent and calyptia-fluentd are bundled and running Ruby versions for now.
This installation guide is for td-agent v3/v4 and calyptia-fluentd v1. td-agent v3/v4 and calyptia-fluentd use fluentd v1 in the core. See this page for the comparison and supported OS.
Installing td-agent
Step 0: Before Installation
Please follow the Pre-installation Guide to configure your OS properly.
Step 1: Install from Apt Repository
NOTE: If your OS is not supported, consider gem installation instead.
NOTE: Treasure Data does not verify Debian packages. If you have any problem with Debian packages, send a patch to fluent-package-builder repository.
A shell script is provided to automate the installation process for each version. The shell script registers a new apt repository at /etc/apt/sources.list.d/treasure-data.list and installs the td-agent deb package.
For Ubuntu Focal:
1
# td-agent 4
2
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-focal-td-agent4.sh | sh
Copied!
For Ubuntu Bionic:
1
# td-agent 4
2
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent4.sh | sh
3
# td-agent 3
4
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh
Copied!
For Ubuntu Xenial:
1
# td-agent 4
2
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent4.sh | sh
3
# td-agent 3
4
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh
Copied!
For Ubuntu Trusty:
1
# td-agent 3
2
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh
Copied!
For Debian Bullseye:
1
# td-agent 4
2
curl -fsSL https://toolbelt.treasuredata.com/sh/install-debian-bullseye-td-agent4.sh | sh
Copied!
For Debian Buster:
1
# td-agent 4
2
curl -fsSL https://toolbelt.treasuredata.com/sh/install-debian-buster-td-agent4.sh | sh
3
# td-agent 3
4
curl -fsSL https://toolbelt.treasuredata.com/sh/install-debian-buster-td-agent3.sh | sh
Copied!
For Debian Stretch:
1
# td-agent 3
2
curl -fsSL https://toolbelt.treasuredata.com/sh/install-debian-stretch-td-agent3.sh | sh
Copied!
For Debian Jessie:
1
# td-agent 3
2
curl -fsSL https://toolbelt.treasuredata.com/sh/install-debian-jessie-td-agent3.sh | sh
Copied!
Step 2: Launch Daemon
systemd
Use /lib/systemd/system/td-agent script to start, stop, or restart the agent:
1
$ sudo systemctl start td-agent.service
2
$ sudo systemctl status td-agent.service
3
● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
4
   Loaded: loaded (/lib/systemd/system/td-agent.service; disabled; vendor preset: enabled)
5
   Active: active (running) since Thu 2017-12-07 15:12:27 PST; 6min ago
6
     Docs: https://docs.treasuredata.com/articles/td-agent
7
  Process: 53192 ExecStart = /opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid (code = exited, statu
8
 Main PID: 53198 (fluentd)
9
   CGroup: /system.slice/td-agent.service
10
           ├─53198 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent
11
           └─53203 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /v
12
​
13
Dec 07 15:12:27 ubuntu systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
14
Dec 07 15:12:27 ubuntu systemd[1]: Started td-agent: Fluentd based data collector for Treasure Data.
Copied!
To customize systemd behavior, put your td-agent.service in /etc/systemd/system.
NOTE: In td-agent 4, path is different. /opt/td-agent/bin instead of /opt/td-agent/embedded/bin
init.d
For non systemd-based system, use /etc/init.d/td-agent script to start, stop, or restart the agent:
1
$ sudo /etc/init.d/td-agent restart
2
$ sudo /etc/init.d/td-agent status
3
td-agent (pid  21678) is running...
Copied!
The following commands are supported:
1
$ sudo /etc/init.d/td-agent start
2
$ sudo /etc/init.d/td-agent stop
3
$ sudo /etc/init.d/td-agent restart
4
$ sudo /etc/init.d/td-agent status
Copied!
Please make sure your configuration file path is:
1
/etc/td-agent/td-agent.conf
Copied!
Step 3: Post Sample Logs via HTTP
The default configuration (/etc/td-agent/td-agent.conf) is to receive logs at an HTTP endpoint and route them to stdout. For td-agent logs, see /var/log/td-agent/td-agent.log.
You can post sample log records with curl command:
1
$ curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test
2
$ tail -n 1 /var/log/td-agent/td-agent.log
3
2018-01-01 17:51:47 -0700 debug.test: {"json":"message"}
Copied!
Using to install calyptia-fluentd
Step 0: Before Installation
Please follow the Pre-installation Guide to configure your OS properly.
Step 1: Install from Apt Repository
NOTE: If your OS is not supported, consider gem installation instead.
A shell script is provided to automate the installation process for each version. The shell script registers a new apt repository at /etc/apt/sources.list.d/calyptia-fluentd.sources and installs the calyptia-fluentd deb package.
For Ubuntu Focal:
1
# calyptia-fluentd 1
2
curl -fsSL https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-ubuntu-focal.sh | sh
Copied!
For Ubuntu Bionic:
1
# calyptia-fluentd 1
2
curl -fsSL https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-ubuntu-bionic.sh | sh
Copied!
For Ubuntu Xenial:
1
# calyptia-fluentd 1
2
curl -fsSL https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-ubuntu-xenial.sh | sh
Copied!
For Debian Buster:
1
# calyptia-fluentd 1
2
curl -fsSL https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-debian-buster.sh | sh
Copied!
Step 2: Launch Daemon
systemd
Use /lib/systemd/system/calyptia-fluentd script to start, stop, or restart the agent:
1
$ sudo systemctl start calyptia-fluentd.service
2
$ sudo systemctl status calyptia-fluentd.service
3
● calyptia-fluentd.service - calyptia-fluentd: Fluentd based data collector for Calyptia Services
4
   Loaded: loaded (/lib/systemd/system/calyptia-fluentd.service; enabled; vendor preset: enabled)
5
   Active: active (running) since Fri 2021-05-28 15:29:45 JST; 1s ago
6
     Docs: https://docs.fluentd.org/
7
  Process: 406739 ExecStart=/opt/calyptia-fluentd/bin/fluentd --log $CALYPTIA_FLUENTD_LOG_FILE --daemon /var/run/calyptia-fluentd/calyptia-fluentd.pid $CALYPTIA_FLUENTD_OPTIONS (code=exited, status=0/SUCCESS)
8
 Main PID: 406762 (fluentd)
9
    Tasks: 5 (limit: 4915)
10
   CGroup: /system.slice/calyptia-fluentd.service
11
           ├─406762 /opt/calyptia-fluentd/bin/ruby /opt/calyptia-fluentd/bin/fluentd --log /var/log/calyptia-fluentd/calyptia-fluentd.log --daemon /var/run/calyptia-fluentd/calyptia-fluentd.pid
12
           └─406835 /opt/calyptia-fluentd/bin/ruby -Eascii-8bit:ascii-8bit /opt/calyptia-fluentd/bin/fluentd --log /var/log/calyptia-fluentd/calyptia-fluentd.log --daemon /var/run/calyptia-fluentd/calyptia-fluentd.pid --under-supervisor
Copied!
To customize systemd behavior, put your calyptia-fluentd.service in /lib/systemd/system.
Please make sure your configuration file path is:
1
/etc/calyptia-fluentd/calyptia-fluentd.conf
Copied!
Step 3: Post Sample Logs via HTTP
The default configuration (/etc/calyptia-fluentd/calyptia-fluentd.conf) is to receive logs at an HTTP endpoint and route them to stdout. For calyptia-fluentd logs, see /var/log/calyptia-fluentd/calyptia-fluentd.log.
You can post sample log records with curl command:
1
$ curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test
2
$ sudo tail -n 1 /var/log/calyptia-fluentd/calyptia-fluentd.log
3
2021-05-28 15:45:17.998214460 +0900 debug.test: {"json":"message"}
Copied!
Next Steps
You are now ready to collect real logs with Fluentd. Refer to the following tutorials on how to collect data from various sources:
Basic Configuration
​Config File​
Application Logs
​Ruby, Java, Python, PHP,
​Perl, Node.js, Scala​
Examples
​Store Apache Log into Amazon S3​
​Store Apache Log into MongoDB​
​Data Collection into HDFS​
For further steps, follow these:
​ChangeLog of td-agent​
​Chef Cookbook​
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.