Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Installation
  • Example Fluentd Configuration
  • Step 1: Counting Incoming Records by Prometheus Filter Plugin
  • Step 2: Counting Outgoing Records by Prometheus Output Plugin
  • Step 3: Expose Metrics by Prometheus Input Plugin via HTTP
  • Check the Configuration
  • Example Prometheus Configuration
  • How to use Prometheus to monitor Fluentd?
  • List of Fluentd Nodes
  • List of Fluentd Metrics
  • Example Prometheus Queries
  • Metrics to Monitor
  • Grafana for Advanced Visualization / Alerting
  • Further Readings

Was this helpful?

  1. Monitoring Fluentd

Monitoring by Prometheus

PreviousOverviewNextMonitoring by REST API

Last updated 3 years ago

Was this helpful?

This article describes how to monitor Fluentd via .

Since both Prometheus and Fluentd are under , Fluentd project is recommending to use Prometheus by default to monitor Fluentd.

Installation

Install fluent-plugin-prometheus gem:

$ fluent-gem install fluent-plugin-prometheus

For td-agent, use td-agent-gem for installation:

$ sudo td-agent-gem install fluent-plugin-prometheus

This contains a fully working configuration for this article.

Example Fluentd Configuration

To expose Fluentd metrics to Prometheus, we need to configure three (3) parts:

  • Step 1: Counting Incoming Records by Prometheus Filter Plugin

  • Step 2: Counting Outgoing Records by Prometheus Output Plugin

  • Step 3: Expose Metrics by Prometheus Input Plugin via HTTP

Step 1: Counting Incoming Records by Prometheus Filter Plugin

Configure the <filter> section to count the incoming records per tag:

# source
<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

# count the number of incoming records per tag
<filter company.*>
  @type prometheus
  <metric>
    name fluentd_input_status_num_records_total
    type counter
    desc The total number of incoming records
    <labels>
      tag ${tag}
      hostname ${hostname}
    </labels>
  </metric>
</filter>

With this configuration, the prometheus filter plugin starts adding the internal counter as the record comes in.

Step 2: Counting Outgoing Records by Prometheus Output Plugin

Configure the copy plugin with prometheus output plugin to count the outgoing records per tag:

# count the number of outgoing records per tag
<match company.*>
  @type copy

  <store>
    @type forward
    <server>
      name myserver1
      host 192.168.1.3
      port 24224
      weight 60
    </server>
  </store>

  <store>
    @type prometheus
    <metric>
      name fluentd_output_status_num_records_total
      type counter
      desc The total number of outgoing records
      <labels>
        tag ${tag}
        hostname ${hostname}
      </labels>
    </metric>
  </store>

</match>

With this configuration, the prometheus output plugin starts adding the internal counter as the record goes out.

Step 3: Expose Metrics by Prometheus Input Plugin via HTTP

Configure prometheus input plugin to expose internal counter information via HTTP:

# expose metrics in prometheus format

<source>
  @type prometheus
  bind 0.0.0.0
  port 24231
  metrics_path /metrics
</source>

<source>
  @type prometheus_output_monitor
  interval 10
  <labels>
    hostname ${hostname}
  </labels>
</source>

Check the Configuration

After you have done these three (3) changes, restart fluentd:

# For stand-alone Fluentd installations
$ fluentd -c fluentd.conf

# For td-agent users
$ sudo systemctl restart td-agent

Let's send some records:

$ echo '{"message":"hello"}' | bundle exec fluent-cat company.test1
$ echo '{"message":"hello"}' | bundle exec fluent-cat company.test1
$ echo '{"message":"hello"}' | bundle exec fluent-cat company.test1
$ echo '{"message":"hello"}' | bundle exec fluent-cat company.test2
curl http://localhost:24231/metrics
# TYPE fluentd_input_status_num_records_total counter
# HELP fluentd_input_status_num_records_total The total number of incoming records
fluentd_input_status_num_records_total{tag="company.test",host="KZK.local"} 3.0
fluentd_input_status_num_records_total{tag="company.test2",host="KZK.local"} 1.0
# TYPE fluentd_output_status_num_records_total counter
# HELP fluentd_output_status_num_records_total The total number of outgoing records
fluentd_output_status_num_records_total{tag="company.test",host="KZK.local"} 3.0
fluentd_output_status_num_records_total{tag="company.test2",host="KZK.local"} 1.0
# TYPE fluentd_output_status_buffer_queue_length gauge
# HELP fluentd_output_status_buffer_queue_length Current buffer queue length.
fluentd_output_status_buffer_queue_length{hostname="KZK.local",plugin_id="object:3fcbccc6d388",type="forward"} 1.0
....

Example Prometheus Configuration

Prepare the configuration file (prometheus.yml):

global:
  scrape_interval: 10s # Set the scrape interval to every 10 seconds. Default is every 1 minute.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'fluentd'
    static_configs:
      - targets: ['localhost:24231']

Launch prometheus:

$ ./prometheus --config.file="prometheus.yml"

Now, open this URL http://localhost:9090/ in your browser.

How to use Prometheus to monitor Fluentd?

List of Fluentd Nodes

Go to http://localhost:9090/targets to see the list of Fluentd nodes and their status.

List of Fluentd Metrics

Visit http://localhost:9090/graph to explore Fluentd's internal metrics. You'll see eight (8) metrics in the metric list:

  • fluentd_input_status_num_records_total

  • fluentd_output_status_buffer_queue_length

  • fluentd_output_status_buffer_total_bytes

  • fluentd_output_status_emit_count

  • fluentd_output_status_num_errors

  • fluentd_output_status_num_records_total

  • fluentd_output_status_retry_count

  • fluentd_output_status_retry_wait

Pick fluentd_input_status_num_records_total and you'll see the total incoming records per tag.

Example Prometheus Queries

Here are the example PromQLs for common metrics:

# number of available nodes
up

# incoming records / sec / host
sum(rate(fluentd_input_status_num_records_total[1m])) by (hostname)

# incoming records / sec / tag
sum(rate(fluentd_input_status_num_records_total[1m])) by (tag)

# outgoing records / sec / host
sum(rate(fluentd_output_status_num_records_total[1m])) by (hostname)

# outgoing records / sec / tag
sum(rate(fluentd_output_status_num_records_total[1m])) by (tag)

# emit count / sec
rate(fluentd_output_status_emit_count[1m])

Metrics to Monitor

In addition to the traffic metrics introduced above, it is important to monitor the queue length and error count.

If these values are increasing, it means Fluentd cannot flush the buffer to the destination. Thus you will lose the data once the buffer becomes full.

# maximum buffer length in last 1min
max_over_time(fluentd_output_status_buffer_queue_length[1m])

# maximum buffer bytes in last 1min
max_over_time(fluentd_output_status_buffer_total_bytes[1m])

# maximum retry wait in last 1min
max_over_time(fluentd_output_status_retry_wait[1m])

# retry count / sec
rate(fluentd_output_status_retry_count[1m])

Grafana for Advanced Visualization / Alerting

Further Readings

Access http://localhost:24231/metrics to receive the metrics in :

Since fluentd_input_status_num_records_total and fluentd_output_status_num_records_total are monotonically increasing numbers, it requires a little bit of calculation by to make them meaningful.

For more advanced visualization and alerting, we recommend as a visualization frontend for Prometheus.

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

Prometheus
CNCF (Cloud Native Computing Foundation)
GitHub repository
Prometheus format
PromQL (Prometheus Query Language)
Grafana
Grafana Support for Prometheus
Prometheus Documentation
Grafana Documentation
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
Prometheus Targets
Prometheus Metrics
Prometheus Graph
Prometheus + Grafana