Versions | v1.0 (td-agent3)

Elasticsearch Output Plugin

The out_elasticsearch Output plugin writes records into Elasticsearch. By default, it creates records by bulk write operaion. This means that when you first import records using the plugin, no record is created immediately.

The record will be created when the chunk_keys condition has been met. To change the output frequency, please specify the time in chunk_keys and specify timekey value in conf.

This document doesn't describe all parameters. If you want to know full features, check the Further Reading section.

Table of Contents

Install

out_elasticsearch is included in td-agent by default (v3.0.1 or later). Fluentd gem users will have to install the fluent-plugin-elasticsearch gem using the following command.

$ fluent-gem install fluent-plugin-elasticsearch

Example Configuration

Additional configurations are optional.

<match my.logs>
  @type elasticsearch
  host localhost
  port 9200
  index_name fluentd
  type_name fluentd
</match>
Please see the Config File article for the basic structure and syntax of the configuration file. For <buffer> section, please check Buffer section cofiguration.

Plugin helpers

Parameters

Common Parameters

@type (required)

The value must be elasticsearch.

host (optional)

The namenode hostname.

port (optional)

The namenode port number.

hosts (optional)

The nodename and its port pairs.

hosts host1:port1,host2:port2,host3:port3
# or
hosts https://customhost.com:443/path,https://username:password@host-failover.com:443

are accepted.

If you specify this option, host and port options are ignored.

user, password, path, scheme, ssl_verify (optional)

user fluent
password fluentd
path /elastic_search/
scheme https

logstash_format (optional)

Defaults to false.

If you specify this option, fluent-plugin-elasticsearch will use logstash-%Y.%m.%d style indices.

index_name (optional)

If you specify this option, fluent-plugin-elasticsearch will use fluentd style index.

@log_level option

The @log_level option allows the user to set different levels of logging for each plugin. The supported log levels are: fatal, error, warn, info, debug, and trace.

Please see the logging article for further details.

Miscellaneous

You can use %{} style placehodlers to escape for URL encoding needed characters.

user %{demo+}
password %{@secret}

are valid configuration.

hosts https://%{j+hn}:%{passw@rd}@host1:443/elastic/,http://host2

are also valid configuration.

But,

user demo+
password @secret

are invalid configuration.

Common Output / Buffer parameters

For common output / buffer parameters, please check the following articles.

Further Reading

Last updated: 2018-02-25 00:05:26 +0000

Versions | v1.0 (td-agent3)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.