Install by DEB Package v3 (Debian/Ubuntu)

This article explains how to install the td-agent deb package, the stable Fluentd distribution package maintained by Treasure Data, Inc.

What is td-agent?

Fluentd is written in Ruby for flexibility, with performance-sensitive parts in C. However, some users may have difficulty installing and operating a Ruby daemon.

That is why Treasure Data, Inc provides the stable distribution of Fluentd, called td-agent. The differences between Fluentd and td-agent can be found here.

Installing td-agent

This article contains deprecated td-agent (EOL) information: SHOULD NOT use td-agent anymore.

• As Treasure Agent (td-agent) 3 will not be maintained anymore, recommend to Upgrade td-agent from v3 to v4.

• Do not directly upgrade from v3 to fluent-package v5. Such a workflow is not supported. It causes a trouble. Upgrade in stages. (v3 to v4, then v4 to v5)

• Package archive was migrated from packages.treasuredata.com to fluentd.cdn.cncf.io. Need to migrate URI in treasure-data.list by yourself or just disable it.

Step 0: Before Installation

Please follow the Pre-installation Guide to configure your OS properly.

Step 1: Install from Apt Repository

NOTE: If your OS is not supported, consider gem installation instead.

A shell script is provided to automate the installation process for each version. The shell script registers a new apt repository at /etc/apt/sources.list.d/treasure-data.list and installs the td-agent deb package.

For Ubuntu Bionic:

# td-agent 3 (EOL)
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh

For Ubuntu Xenial:

# td-agent 3 (EOL)
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh

For Debian Buster:

# td-agent 3 (EOL)
curl -fsSL https://toolbelt.treasuredata.com/sh/install-debian-buster-td-agent3.sh | sh

Step 2: Launch Daemon

systemd

Use /lib/systemd/system/td-agent script to start, stop, or restart the agent:

$ sudo systemctl start td-agent.service
$ sudo systemctl status td-agent.service
● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
    Loaded: loaded (/lib/systemd/system/td-agent.service; enabled; vendor preset: enabled)
    Active: active (running) since Tue 2022-09-20 05:28:25 JST; 1 week 0 days ago
      Docs: https://docs.treasuredata.com/display/public/PD/About+Treasure+Data%27s+Server-Side+Agent
  Main PID: 2417 (fluentd)
    Tasks: 8 (limit: 38328)
    Memory: 52.2M
      CPU: 56.868s
    CGroup: /system.slice/td-agent.service
            ├─2417 /opt/td-agent/bin/ruby /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
            └─2420 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/t>

Sept 9 20 05:28:24 Ryzen systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
Sept 9 20 05:28:25 Ryzen systemd[1]: Started td-agent: Fluentd based data collector for Treasure Data.

To customize systemd behavior, put your td-agent.service in /etc/systemd/system.

NOTE: In td-agent 4, path is different. /opt/td-agent/bin instead of /opt/td-agent/embedded/bin

init.d

For non systemd-based system, use /etc/init.d/td-agent script to start, stop, or restart the agent:

$ sudo /etc/init.d/td-agent restart
$ sudo /etc/init.d/td-agent status
td-agent (pid  21678) is running...

The following commands are supported:

$ sudo /etc/init.d/td-agent start
$ sudo /etc/init.d/td-agent stop
$ sudo /etc/init.d/td-agent restart
$ sudo /etc/init.d/td-agent status

Please make sure your configuration file path is:

/etc/td-agent/td-agent.conf

Step 3: Post Sample Logs via HTTP

The default configuration (/etc/td-agent/td-agent.conf) is to receive logs at an HTTP endpoint and route them to stdout. For td-agent logs, see /var/log/td-agent/td-agent.log.

You can post sample log records with curl command:

$ curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test
$ tail -n 1 /var/log/td-agent/td-agent.log
2018-01-01 17:51:47 -0700 debug.test: {"json":"message"}

Next Steps

You are now ready to collect real logs with Fluentd. Refer to the following tutorials on how to collect data from various sources:

• Basic Configuration

  • Config File

• Application Logs

  • Ruby, Java, Python, PHP,

    Perl, Node.js, Scala

• Examples

  • Store Apache Log into Amazon S3

  • Store Apache Log into MongoDB

  • Data Collection into HDFS

For further steps, follow these:

• ChangeLog of td-agent

• Chef Cookbook

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.