Python

The 'fluent-logger-python', library is used to post records from Python applications to Fluentd.

This article explains how to use the fluent-logger-python library.

Prerequisites

• Basic knowledge of Python

• Basic knowledge of Fluentd

• Python 2.6 or higher

Installing Fluentd

Please refer to the following documents to install fluentd.

• Install Fluentd with rpm Package

• Install Fluentd with deb Package

• Install Fluentd with Ruby Gem

• Install Fluentd from source

Modifying the Config File

Next, please configure Fluentd to use the forward Input plugin as its data source.

<source>
  @type forward
  port 24224
</source>
<match fluentd.test.**>
  @type stdout
</match>

Please restart your agent once these lines are in place.

# for rpm/deb only
$ sudo /etc/init.d/td-agent restart

Using fluent-logger-python

First, install the fluent-logger library via pip.

$ pip install fluent-logger

Next, initialize and post the records as shown below.

# test.py
from fluent import sender
from fluent import event
sender.setup('fluentd.test', host='localhost', port=24224)
event.Event('follow', {
  'from': 'userA',
  'to':   'userB'
})

Executing the script will send the logs to Fluentd.

$ python test.py

The logs should be output to /var/log/td-agent/td-agent.log or stdout of the Fluentd process via the stdout Output plugin.

Production Deployments

Output Plugins

Various output plugins are available for writing records to other destinations:

• Examples

  • Store Apache Logs into Amazon S3

  • Store Apache Logs into MongoDB

  • Data Collection into HDFS

• List of Plugin References

  • Output to Another Fluentd

  • Output to MongoDB or MongoDB ReplicaSet

  • Output to Hadoop

  • Output to File

  • etc...

High-Availability Configurations of Fluentd

For high-traffic websites (more than 5 application nodes), we recommend using a high availability configuration of td-agent. This will improve data transfer reliability and query performance.

• High-Availability Configurations of Fluentd

Monitoring

Monitoring Fluentd itself is also important. The article below describes general monitoring methods for td-agent.

• Monitoring Fluentd

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.