Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Example Configuration
  • Parameters
  • @type (required)
  • port
  • bind
  • tag
  • emit_interval
  • include_config
  • include_retry
  • Configuration Example
  • Output Example
  • retry
  • Tips and Tricks
  • How to use query parameters to tune outputs
  • How to emit metrics as events
  • Multi-Process Environment

Was this helpful?

  1. Input Plugins

monitor_agent

PrevioussampleNextwindows_eventlog

Last updated 2 years ago

Was this helpful?

The in_monitor_agent Input plugin exports Fluentd's internal metrics via REST API.

It is included in Fluentd's core.

Example Configuration

<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
</source>

This configuration launches HTTP server with 24220 port and get metrics:

$ curl http://host:24220/api/plugins.json

Also you can fetch the same data in LTSV format:

$ curl http://host:24220/api/plugins

Refer to the article for the basic structure and syntax of the configuration file.

Parameters

See .

@type (required)

The value must be monitor_agent.

port

type
default
version

integer

24220

0.14.0

The port to listen to.

bind

type
default
version

string

0.0.0.0 (all addresses)

0.14.0

The bind address to listen to.

tag

type
default
version

string

nil

0.14.0

If you set this parameter, this plugin emits metrics as records. See "Reuse plugins" section.

emit_interval

type
default
version

integer

60

0.14.0

The interval time between event emits. This will be used when tag is configured.

include_config

type
default
version

bool

true

0.14.0

You can set this option to false to remove the config field from the response.

include_retry

type
default
version

bool

true

0.14.11

You can set this option to false to remove the retry field from the response.

Configuration Example

Here is a configuration example using in_monitor_agent:

<source>
  @type monitor_agent
  @id in_monitor_agent
  include_config false
</source>

<source>
  @type forward
  @id in_forward
</source>

<match test.**>
  @type elasticsearch
  @id out_es
</match>

When using this plugin, we strongly recommend setting @id on each plugin in use. This makes the task to identify which record corresponds to which plugin much easier. Without @id, Fluentd uses object_id as the unique identifier, so you cannot identify a record just by looking at its plugin_id field.

Output Example

Here is how the output looks like in JSON:

{
  "plugins": [
    {
      "plugin_id": "in_monitor_agent",
      "plugin_category": "input",
      "type": "monitor_agent",
      "output_plugin": false,
      "retry_count": null
    },
    {
      "plugin_id": "in_forward",
      "plugin_category": "input",
      "type": "forward",
      "output_plugin": false,
      "retry_count": null
    },
    {
      "plugin_id": "out_es",
      "plugin_category": "output",
      "type": "elasticsearch",
      "output_plugin": true,
      "buffer_queue_length": 0,
      "buffer_total_queued_size": 0,
      "retry_count": 0,
      "retry": {}
    }
  ]
}

If the plugin is an output plugin with the buffer settings, the metrics include the buffer related fields.

retry

If the output plugin is in retry status, additional fields are added to retry. For example, if the Elasticsearch plugin fails to flush the buffer.

Here is the response:

{ 
  "plugin_id": "out_es",
  "plugin_category": "output",
  "type": "elasticsearch",
  "output_plugin": true,
  "buffer_queue_length": 1,
  "buffer_total_queued_size": 0,
  "retry_count": 3,
  "retry": {
    "start": "2018-01-30 22:42:47 +0900",
    "steps": 2,
    "next_time": "2018-01-30 22:42:52 +0900"
  }
}

steps field in retry show the number of flush failures, so next is the third try. retry_count is the total number of flush failures. This value is cleared when fluentd restarts, not when retry succeeds.

Tips and Tricks

How to use query parameters to tune outputs

This plugin supports a number of query parameters with which you can customize the output format of HTTP responses. For example, you can append debug=1 to the request URL to get the verbose internal metrics:

$ curl http://localhost:24220/api/plugins.json?debug=1

The following list shows the available query parameters:

Parameter
Value
Explanation

debug

Constant

Expose additional internal metrics. Note that sensitive configuration value (password and so on) will be shown without obfuscating

with_ivars

Variable names

Expose the specified instance variables of each plugin

with_config

Boolean

Override the configuration option with_config

with_retry

Boolean

Override the configuration option with_retry

tag

Event tag

Only show plugins that matches the specified tag

@id

Plugin id

Filter plugins by plugin id

@type

Plugin type

Filter plugins by plugin type

How to emit metrics as events

You can emit the internal metrics as events by setting the tag.

For example:

<source>
  @type monitor_agent
  tag debug.monitor
  emit_interval 60
  port 24230
</source>

Note that in_monitor_agent produces separate records for each plugin. Thus, using this configuration, you will receive events like below once per minute:

2018-01-30 22:53:29.591560000 +0900 debug.monitor: { "plugin_id":"object:3ffd9988bea0","plugin_category":"input","type":"monitor_agent","output_plugin":false,"retry_count":null}
2018-01-30 22:53:29.591560000 +0900 debug.monitor: { "plugin_id":"in_forward","plugin_category":"input","type":"forward","output_plugin":false,"retry_count":null}
2018-01-30 22:53:29.591560000 +0900 debug.monitor: { "plugin_id":"out_out","plugin_category":"output","type":"stdout","output_plugin":true,"retry_count":0}

Multi-Process Environment

If you use this plugin under the multi-process environment, the HTTP server will be launched in each worker. Port is assigned sequentially. For example, with this configuration:

<system>
  workers 3
</system>

<source>
  @type monitor_agent
  port 24230
</source>

Three (3) HTTP servers will be launched with:

  • port 24230 for worker 0

  • port 24231 for worker 1

  • port 24232 for worker 2

Note that you may need to set worker_id to @id parameter. See .

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

Configuration File
Common Parameters
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
config article