Versions | v1.0 (td-agent3)

Monitoring Agent Input Plugin

The in_monitor_agent Input plugin enables Fluentd to export internal metrics by using HTTP API.

Table of Contents

Example Configuration

in_monitor_agent is included in Fluentd’s core. No additional installation process is required.

<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
</source>

This configuration launches HTTP server with 24220 port and get metrics like below:

$ curl http://host:24220/api/plugins.json

Also you can fetch the same data in LTSV format:

$ curl http://host:24220/api/plugins
Please see the Config File article for the basic structure and syntax of the configuration file.

Parameters

Common Parameters

@type (required)

The value must be monitor_agent.

port

type default version
integer 24220 0.14.0

The port to listen to.

bind

type default version
string 0.0.0.0 (all addresses) 0.14.0

The bind address to listen to.

tag

type default version
string nil 0.14.0

If you set this parameter, this plugin emits metrics as records. See “Reuse plugins” section.

emit_interval

type default version
integer 60 0.14.0

The interval time between event emits. This will be used when tag is configured.

include_config

type default version
bool true 0.14.0

You can set this option to false to remove config field from the response.

include_retry

type default version
bool true 0.14.11

You can set this option to false to remove retry field from the response.

Configuration Example

Here is a configuration example that uses in_monitor_agent.

<source>
  @type monitor_agent
  @id in_monitor_agent
  include_config false
</source>

<source>
  @type forward
  @id in_forward
</source>

<match test.**>
  @type elasticsearch
  @id out_es
</match>

When using this plugin, we strongly recommend to set @id on each plugin in use. This makes the task to identify which record corresponds to which plugin much easier (Without @id, Fluentd uses object_id as unique identifier, so you cannot identify a record just by looking at its plugin_id field)

Output Example

Here is how the output looks like (in JSON format):

{
  "plugins": [
    {
      "plugin_id": "in_monitor_agent",
      "plugin_category": "input",
      "type": "monitor_agent",
      "output_plugin": false,
      "retry_count": null
    },
    {
      "plugin_id": "in_forward",
      "plugin_category": "input",
      "type": "forward",
      "output_plugin": false,
      "retry_count": null
    },
    {
      "plugin_id": "out_es",
      "plugin_category": "output",
      "type": "elasticsearch",
      "output_plugin": true,
      "buffer_queue_length": 0,
      "buffer_total_queued_size": 0,
      "retry_count": 0,
      "retry": {}
    }
  ]
}

If the plugin is an output with buffer setting, the metrics has buffer related fields.

In retry

If the output plugin is in retry status, additional fields are added to retry. For example, if the elasticsearch plugin fails to flush the buffer, response is below:

{ 
  "plugin_id": "out_es",
  "plugin_category": "output",
  "type": "elasticsearch",
  "output_plugin": true,
  "buffer_queue_length": 1,
  "buffer_total_queued_size": 0,
  "retry_count": 3,
  "retry": {
    "start": "2018-01-30 22:42:47 +0900",
    "steps": 2,
    "next_time": "2018-01-30 22:42:52 +0900"
  }
}

steps field in retry show the number of flush failure, so next is 3rd try. retry_count is the total number of flush failure. This value is cleared when fluentd restart, not retry succeeded.

Tips & Tricks

How to use query parameters to tune outputs

This plugin supports a number of query parameters with which you can customize the output format of HTTP responses. For example, you can append debug=1 to the request URL to get the verbose internal metrics.

$ curl http://localhost:24220/api/plugins.json?debug=1

The following list shows the available query parameters.

Parameter Value Explanation
debug Constant Expose additional internal metrics
with_ivars Variable names Expose the specified instance variables of each plugin
with_config Boolean Override the configuration option with_config
with_retry Boolean Override the configuration option with_retry
tag Event tag Only show plugins that matches the specified tag
@id Plugin id Filter plugins by plugin id
@type Plugin type Filter plugins by plugin type

How to emit metrics as events

You can emits the internal metrics as events by setting the option tag. For example:

<source>
  @type monitor_agent
  tag debug.monitor
  emit_interval 60
  port 24230
</source>

Note that in_monitor_agent produces separate records for each plugin. Thus, using this configuration, you will receive events like below once per minute.

2018-01-30 22:53:29.591560000 +0900 debug.monitor: { "plugin_id":"object:3ffd9988bea0","plugin_category":"input","type":"monitor_agent","output_plugin":false,"retry_count":null}
2018-01-30 22:53:29.591560000 +0900 debug.monitor: { "plugin_id":"in_forward","plugin_category":"input","type":"forward","output_plugin":false,"retry_count":null}
2018-01-30 22:53:29.591560000 +0900 debug.monitor: { "plugin_id":"out_out","plugin_category":"output","type":"stdout","output_plugin":true,"retry_count":0}

Multi-process environment

If you use this plugin under multi-process environment, HTTP server will be launched in each worker. Port is assigned in sequential number. For example, with this configuration:

<system>
  workers 3
</system>

<source>
  @type monitor_agent
  port 24230
</source>

3 HTTP servers will be launched. Port 24230 for worker 0, port 24231 for worker 1 and port 24232 for worker 2. Note that you may need to set worker_id to @id parameter. See config article

Last updated: 2018-11-18 18:37:07 +0000

Versions | v1.0 (td-agent3)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.