Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • System Administration
  • For fluent-package
  • For td-agent
  • For calyptia-fluentd
  • Connect to the Other Services
  • How It Works
  • Plugin Management
  • Available Plugins
  • Configuration Syntax
  • Data Source
  • Output Endpoint
  • Further Reading

Was this helpful?

  1. Installation

Post Installation Guide

PreviousInstall from SourceNextObsolete Installation

Last updated 4 months ago

Was this helpful?

This article discusses the post-installation steps for new Fluentd users assuming that Fluentd has been installed using the fluent-package, td-agent and calyptia-fluentd package.

System Administration

For fluent-package

Configuration File

After successful installation, a fluent-package instance will be up and running with a predefined template configuration file.

The default path for this configuration file is:

/etc/fluent/fluentd.conf

You may edit this configuration file according to your own use case.

After editing the configuration file, restart fluentd using systemctl command:

$ sudo systemctl restart fluentd

Logging

By default, Fluentd writes its operation logs to the following file:

/var/log/fluent/fluentd.log

For more verbose logs, read the article on .

For td-agent

As td-agent had reached EOL, SHOULD NOT use td-agent anymore.

Configuration File

After successful installation, a td-agent instance will be up and running with a predefined template configuration file.

The default path for this configuration file is:

/etc/td-agent/td-agent.conf

You may edit this configuration file according to your own use case.

After editing the configuration file, restart td-agent using systemctl command:

$ sudo systemctl restart td-agent

Logging

By default, td-agent writes its operation logs to the following file:

/var/log/td-agent/td-agent.log

For calyptia-fluentd

Configuration File

After the successful installation, a calyptia-fluentd instance will be up and running with a predefined template configuration file.

The default path for this configuration file is:

/etc/calyptia-fluentd/calyptia-fluentd.conf

You may edit this configuration file according to your own use case.

After editing the configuration file, restart calyptia-fluentd using systemctl command:

$ sudo systemctl restart calyptia-fluentd

Logging

By default, calyptia-fluentd writes its operation logs to the following file:

/var/log/calyptia-fluentd/calyptia-fluentd.log

Connect to the Other Services

How It Works

In Fluentd, the most important part of data input/output is managed by plugins. Each plugin knows how to interface with an external endpoint and is solely responsible for managing one pipeline to forward data streams.

Plugins use a certain naming convention. For example, if it receives data and interfaces with Apache Kafka, it is called in_kafka. In the same way, if it publishes data and connects to MongoDB, it is called out_mongo.

The following configuration uses the in_forward plugin as an input source and out_file plugin as an output endpoint:

<source>
  @type forward
  port 9999
</source>
<match app.**>
  @type file
  path /var/log/app/data.log
  compress gzip
</match>

Plugin Management

Fluentd manages plugins as Ruby gems but stores them in their dedicated directory separated from the normal Ruby gems.

fluent-package

A special program fluent-gem is used to manage plugin gems. For example, the following command installs a plugin to connect to S3 (including both in_s3 and out_s3 plugins):

 $ sudo fluent-gem install fluent-plugin-s3

td-agent

As td-agent had reached EOL, SHOULD NOT use td-agent anymore.

A special program td-agent-gem is used to manage plugin gems. For example, the following command installs a plugin to connect to S3 (including both in_s3 and out_s3 plugins):

 $ sudo /usr/sbin/td-agent-gem install fluent-plugin-s3

calyptia-fluentd

A special program calyptia-fluentd-gem is used to manage plugin gems. For example, the following command installs a plugin to connect to S3 (including both in_s3 and out_s3 plugins):

 $ sudo /usr/sbin/calyptia-fluentd-gem install fluent-plugin-s3

Available Plugins

Note that a number of plugins are bundled with the standard distribution of td-agent so you do not need to install them manually.

Configuration Syntax

Data Source

A configuration file consists of a number of setting blocks or sections e.g. <source>. Each block contains a set of options for a specific data endpoint.

<source>
  @type syslog
  port 5140
  tag system
</source>

The @type parameter specifies which plugin to use. Note that the plugin type prefix i.e. in_, out_, etc. is not needed here. In this example, the input plugin is specified as syslog, not in_syslog.

Output Endpoint

To add an output endpoint for the data stream, you need to define a <match> block. Syntactically, <match> is slightly different from <source> in the sense that it requires a filter expression as an argument.

For example, if you want to output events tagged with debug.log, you need to mention this tag as an argument in <match> like this:

<match debug.log>
  @type kafka2
  port 5140
  brokers kafka-server:9092
  tag system
  # ...
</match>

The wildcard character * can be used in the filter expression. For example, debug.* matches debug.log, debug.foo, etc.

To catch all the descendent tags, use double asterisks **. For example, debug.** matches not only debug.log, but also debug.log.bar or debug.log.level.critical, etc.

Further Reading

Recommend to .

For more verbose logs, read the article on .

For more verbose logs, read the article on .

Recommend to .

See the to explore the available third-party plugins.

For example, if you want to create an endpoint to receive data from , you need to add a <source> block and set up its settings like this:

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

Troubleshooting
Upgrade to fluent-package v5
Troubleshooting
Troubleshooting
Upgrade to fluent-package v5
List Of All Plugins
syslog
Configuration File Syntax
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)