http - Fluentd

Search…
http
The out_http Output plugin writes records via HTTP/HTTPS.
This plugin is introduced since fluentd v1.7.0.
It is included in Fluentd's core.
Example Configuration
1
<match pattern>
2
  @type http
3
​
4
  endpoint http://logserver.com:9000/api
5
  open_timeout 2
6
​
7
  <format>
8
    @type json
9
  </format>
10
  <buffer>
11
    flush_interval 10s
12
  </buffer>
13
</match>
Copied!
Please see the Configuration File article for the basic structure and syntax of the configuration file.
For <buffer>, refer to Buffer Section Configuration.
Parameters
@type
The value must be http.
endpoint
type
default
version
string
required
1.7.0
The endpoint for HTTP request. If you want to use HTTPS, use https prefix.
1
# Use HTTP
2
endpoint http://example.com/api
3
​
4
# Use HTTPS. You can set additional HTTPS parameters like tls_xxx
5
endpoint https://example.com/api
Copied!
The endpoint parameter supports placeholders, so you can embed time, tag and record fields in the endpoint. Placeholders also require the buffer section in order to work. Here is an example:
1
endpoint http://example.com/api/${tag}-${key}
2
<buffer tag,key>
3
  # buffer parameters
4
</buffer>
Copied!
See Buffer Section Configurations for more details.
http_method
type
default
available values
version
enum
post
post/put
1.7.0
The method for HTTP request.
proxy
type
default
version
string
optional
1.7.0
The proxy for HTTP request.
content_type
type
default
version
string
nil
1.7.0
Content-Type for HTTP request. out_http automatically set Content-Type for built-in formatters when this parameter is not specified.
Here is a table:
json with json_array false: application/x-ndjson
json with json_array true: application/json
csv: text/csv
tsv, ltsv: text/tab-separated-values
msgpack: application/x-msgpack
out_file, single_value, stdout, hash: text/plain
json_array
type
default
version
bool
false
1.10.4
Using the array format of JSON. This parameter is used and valid only for json format. When json_array as true, Content-Type should be application/json and be able to use JSON data for the HTTP request body.
json_array true
1
[
2
  {"key":"value", ...},
3
  {"key":"value", ...},
4
  ...
5
]
Copied!
json_array false
1
{"key":"value", ...}
2
{"key":"value", ...}
3
...
Copied!
<format> Directive
The format of the payload. The default @type is json.
Here is single_value example:
1
<format>
2
  @type single_value
3
</format>
Copied!
See formatter article for more detail.
headers
type
default
version
hash
nil
1.7.0
Additional headers for HTTP request.
1
headers {"key1":"value1", "key2":"value2"}
Copied!
headers_from_placeholders
type
default
version
hash
nil
1.12.1
Additional placeholder based headers for HTTP request. If you want to use tag or record field, use this parameter instead of headers.
1
headers_from_placeholders {"x-foo-bar":"${$.foo.bar}","x-tag":"app-${tag}"}
2
<buffer tag,$.foo.bar>
3
  # buffer parameters...
4
</buffer>
Copied!
open_timeout
type
default
version
integer
nil
1.7.0
The connection open timeout in seconds.
See also Ruby document.
read_timeout
type
default
version
integer
nil
1.7.0
The read timeout in seconds.
See also Ruby document.
ssl_timeout
type
default
version
integer
nil
1.7.0
The TLS timeout in seconds.
tls_ca_cert_path
type
default
version
string
nil
1.7.0
The CA certificate path for TLS.
tls_client_cert_path
type
default
version
string
nil
1.7.0
The client certificate path for TLS.
tls_private_key_path
type
default
version
string
nil
1.7.0
The client private key path for TLS.
tls_private_key_passphrase
type
default
version
string
nil
1.7.0
The client private key passphrase for TLS.
tls_verify_mode
type
default
available values
version
enum
peer
peer/none
1.7.0
The verify mode of TLS.
tls_version
type
default
available values
version
enum
TLSv1_2
TLSv1_2/TLSv1_1
1.7.0
The default version of TLS.
tls_ciphers
type
default
version
string
ALL:!aNULL:!eNULL:!SSLv2
1.7.0
The cipher suites configuration of TLS.
error_response_as_unrecoverable
type
default
version
bool
true
1.7.0
Raise UnrecoverableError when the response code is not SUCCESS, 1xx/3xx/4xx/5xx. If false, out_http logs error message instead of raising UnrecoverableError.
See also Handling Unrecoverable Errors.
retryable_response_codes
type
default
version
array of int
[503]
1.7.0
The list of retryable response codes. If the response code is included in this list, out_http retries the buffer flush.
<auth> Section
Specifies HTTP authentication:
1
<auth>
2
  method basic
3
  username fluentd
4
  password hello
5
</auth>
Copied!
method
type
default
available values
version
enum
basic
basic
1.7.0
The method for HTTP authentication. Now only basic.
username
type
default
version
string
nil
1.7.0
The username for basic authentication.
password
type
default
version
string
nil
1.7.0
The password for basic authentication.
Common Output / Buffer parameters
For common output / buffer parameters, please check the following articles:
​Output Plugin Overview​
​Buffer Section Configuration​
The Payload Content
out_http's request body depends on <format> configuration. For example, the default setting generates newline delimited JSON like this:
1
# \n is added by `add_newline true` parameter in <format>
2
{"key1":"value1","key2":"value2",...}\n
3
{"key1":"value1","key2":"value2",...}\n
4
{"key1":"value1","key2":"value2",...}\n
5
{"key1":"value1","key2":"value2",...}\n
6
...
Copied!
With @type single_value:
1
log line1\n
2
log line2\n
3
log line3\n
4
log line4\n
5
...
Copied!
Troubleshooting
400 Bad request between out_http and in_http
When getting the following error:
1
#0 got unrecoverable error in primary and no secondary error_class=Fluent::UnrecoverableError error="400 Bad Request 400 Bad Request\n'json' or 'msgpack' parameter is required\n"
2
#0 bad chunk is moved to /tmp/fluent/backup/worker0/object_3ff8a73edae8/5a71a08ca19b1b343c8dce1b74c9a963.log
Copied!
Users should specify json format with json_array as true for out_http configuration:
1
<match **>
2
  @type http
3
  endpoint http://some.your.http.endpoint:9811/your-awesome-path
4
  <format>
5
    @type json
6
  </format>
7
  json_array true
8
  <buffer>
9
    flush_interval 2s
10
  </buffer>
11
</match>
Copied!
And receiver in_http configuration should be:
1
<source>
2
  @type http
3
  port 9811
4
  bind 0.0.0.0
5
  <parse>
6
    @type json
7
  </parse>
8
</source>
Copied!
Or specify msgpack format:
1
<match **>
2
  @type http
3
  endpoint http://some.your.http.endpoint:9882/your-awesome-path
4
  <format>
5
    @type msgpack
6
  </format>
7
  <buffer>
8
    flush_interval 2s
9
  </buffer>
10
</match>
Copied!
And, receiver in_http configuration should be:
1
<source>
2
  @type http
3
  port 9882
4
  bind 0.0.0.0
5
  <parse>
6
    @type msgpack
7
  </parse>
8
  <format>
9
    @type json
10
  </format>
11
</source>
Copied!
But, we recommend to use in/out forward plugin to communicate with two Fluentd instances due to at-most-once and at-least-once semantics for rigidity.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open article- source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
forward
exec
Last modified 7mo ago
Example Configuration

Parameters

`@type`

`endpoint`

`http_method`

`proxy`

`content_type`

`json_array`

`<format>` Directive

`headers`

headers_from_placeholders

`open_timeout`

`read_timeout`

`ssl_timeout`

`tls_ca_cert_path`

`tls_client_cert_path`

`tls_private_key_path`

`tls_private_key_passphrase`

`tls_verify_mode`

`tls_version`

`tls_ciphers`

`error_response_as_unrecoverable`

`retryable_response_codes`

`<auth>` Section

`method`

`username`

`password`

Common Output / Buffer parameters

The Payload Content

Troubleshooting

400 Bad request between `out_http` and `in_http`
