Fluentd
Search…
1.0
Introduction
Overview
Installation
Configuration
Deployment
Container Deployment
Monitoring Fluentd
Input Plugins
Output Plugins
file
forward
http
exec
exec_filter
secondary_file
copy
relabel
roundrobin
stdout
null
s3
kafka
elasticsearch
opensearch
mongo
mongo_replset
rewrite_tag_filter
webhdfs
Filter Plugins
Parser Plugins
Formatter Plugins
Buffer Plugins
Storage Plugins
Service Discovery Plugins
Metrics Plugins
How-to Guides
Language Bindings
Plugin Development
Plugin Helper API
Troubleshooting Guide
Powered By GitBook
opensearch
The out_opensearch Output plugin writes records into OpenSearch. By default, it creates records using bulk api which performs multiple indexing operations in a single API call. This reduces overhead and can greatly increase indexing speed. This means that when you first import records using the plugin, records are not immediately pushed to OpenSearch.
Records will be sent to OpenSearch when the chunk_keys condition has been met. To change the output frequency, please specify the time in chunk_keys and specify timekey value in the configuration.
This document does not describe all the parameters. For details, refer to the Further Reading section.

Installation

Since out_opensearch has been included in the alternative distribution of calyptia-fluentd since v1.3.4, calyptia-fluentd users do not need to install it manually.
If you have installed Fluentd without calyptia-fluentd, please install this plugin using fluent-gem or td-agent-gem (for td-agent users):
$ fluent-gem install fluent-plugin-opensearch
$ td-agent-gem install fluent-plugin-opensearch

Example Configuration

Here is a simple working configuration for OpenSearch instance that is running on localhost:
<match my.logs>
@type opensearch
host localhost
port 9200
logstash_format true
</match>
For more details on each option, read the section on Parameters.

Plugin Helpers

Parameters

@type (required)

This option must be always opensearch.

host (optional)

The hostname of your OpenSearch node (default: localhost).

port (optional)

The port number of your OpenSearch node (default: 9200).

hosts (optional)

If you want to connect to more than one OpenSearch nodes, specify this option in the following format:
hosts host1:port1,host2:port2,host3:port3
# or
hosts https://customhost.com:443/path,https://username:[email protected]:443
If you use this option, the host and port options are ignored.

user, password (optional)

The login credentials to connect to the OpenSearch node (default: nil):
user fluent
password mysecret

scheme (optional)

Specify https if your OpenSearch endpoint supports SSL (default: http).

path (optional)

The REST API endpoint of OpenSearch to post write requests (default: nil).

index_name (optional)

The index name to write events to (default: fluentd).
This option supports the placeholder syntax of Fluentd plugin API. For example, if you want to partition the index by tags, you can specify it like this:
index_name fluentd.${tag}
Here is a more practical example which partitions the OpenSearch index by tags and timestamps:
index_name fluentd.${tag}.%Y%m%d
Time placeholder needs to set up tag and time in chunk_keys. Also, it needs to specify timekey for time slice of chunk:
<buffer tag, time>
timekey 1h # chunks per hours ("3600" also available)
</buffer>
For more information about buffer options checkout the Buffer Section Configuration.

logstash_format (optional)

If true, Fluentd uses the conventional index name format logstash-%Y.%m.%d (default: false). This option supersedes the index_name option.

@log_level option

The @log_level option allows the user to set different levels of logging for each plugin.
Supported log levels: fatal, error, warn, info, debug, trace.
Please see the logging article for further details.

logstash_prefix (optional)

The logstash prefix index name to write events when logstash_format is true (default: logstash).

Miscellaneous

You can use %{} style placeholders to escape for URL encoding needed characters.
Valid configuration:
user %{demo+}
password %{@secret}
Valid configuration:
hosts https://%{j+hn}:%{[email protected]}@host1:443/elastic/,http://host2
Invalid configuration:
user demo+
password @secret

Common Output / Buffer parameters

For common output / buffer parameters, please check the following articles:

Troubleshooting

Please refer to the OpenSearch's troubleshooting section.

Further Reading

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Previous
elasticsearch
Next
mongo
Last modified 7mo ago
Copy link
On this page
Installation
Example Configuration
Plugin Helpers
Parameters
@type (required)
host (optional)
port (optional)
hosts (optional)
user, password (optional)
scheme (optional)
path (optional)
index_name (optional)
logstash_format (optional)
logstash_prefix (optional)
Miscellaneous
Common Output / Buffer parameters
Troubleshooting
Further Reading