Versions | v1.0 (td-agent3) | v0.12 (td-agent2)

HTTP Input Plugin

The in_http Input plugin allows clients to send data in the form of HTTP requests. This is a very handy plugin since you can talk to a Fluentd server in much the same way as to a REST API endpoint.

Table of Contents

Configuration

The following snippet shows an example configuration.

<source>
  @type http
  port 9880
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
</source>

For the full list of the configurable options, see the “Parameters” section.

Basic Usage

Here is a simple example to post a record using curl.

# Post a record with the tag "app.log"
$ curl -X POST -d 'json={"foo":"bar"}' http://localhost:9880/app.log

By default, timestamps are assigned to each record on arrival. You can override the timestamp using the time parameter.

# Overwrite the timestamp to 2018-02-16 04:40:37.3137116
$ curl -X POST -d 'json={"foo":"bar"}' \
  http://localhost:9880/test.tag?time=1518756037.3137116

For more advanced usage, please read the “Tips & Tricks” section.

Parameters

Common Parameters

@type (required)

The value must be http.

port

type default version
integer 9880 0.14.0

The port to listen to.

bind

type default version
string 0.0.0.0 (all addresses) 0.14.0

The bind address to listen to.

body_size_limit

type default version
size 32MB 0.14.0

The size limit of the POSTed element.

keepalive_timeout

type default version
size 10 (seconds) 0.14.0

The timeout limit for keeping the connection alive.

add_http_headers

type default version
bool false 0.14.0

Add HTTP_ prefix headers to the record.

add_remote_addr

type default version
bool false 0.14.0

Add REMOTE_ADDR field to the record. The value of REMOTE_ADDR is the client’s address.

If your system set multiple X-Forwarded-For headers in the request, in_http uses first one. For example:

X-Forwarded-For: host1, host2
X-Forwarded-For: host3

If send above multiple headers, REMOTE_ADDR value is host1.

cors_allow_origins

type default version
array nil(disabled) 0.14.0

White list domains for CORS.

If you set ["domain1", "domain2"] to cors_allow_origins, in_http returns 403 to access from othe domains.

format (deprecated)

Deprecated parameter. Use the <parse> directive as explained below instead.

Tips & Tricks

Send data in MessagePack format

You can post a record in MessagePack format by adding the msgpack= prefix.

# Send data in msgpack format
$ msgpack=`echo -e "\x81\xa3foo\xa3bar"`
$ curl -X POST -d "msgpack=$msgpack" http://localhost:9880/app.log

This reduces the overhead of parsing JSON, hence is relatively faster.

Leverage HTTP Content-Type header

This plugin recognizes the ‘Content-Type’ header in a POST request. Using this header, you can send a JSON payload without adding the json= prefix.

# Post a raw JSON data using Content-Type
$ curl -X POST -d '{"foo":"bar"}' -H 'Content-Type: application/json' \
  http://localhost:9880/app.log

To use MessagePack, set the content type to application/msgpack.

$ msgpack=`echo -e "\x81\xa3foo\xa3bar"`
$ curl -X POST -d "$msgpack" -H 'Content-Type: application/msgpack' \
  http://localhost:9880/app.log

Handle various formats using parser plugins

You can handle various input formats (other than JSON and MessagePack) by using the <parser> directive. For example, add the following settings to the configuration file:

<source>
  @type http
  port 9880
  <parse>
    @type regexp
    expression /^(?<field1>\d+):(?<field2>\w+)$/
  </parse>
</source>

Now you can post custom-format records as below:

# This will be parsed into {"field1":"123456","field2":"awesome"}
$ curl -X POST -d '123456:awesome' http://localhost:9880/app.log

Many other formats (e.g. csv/syslog/nginx) are also supported as well. You can find the full list of supported formats in this article.

Note that parser plugins do not support the batch mode. So if you want to use bulk insertion for handling a large data set, please consider to keep using the default JSON (or MessagePack) format.

Enhance Performance

Handle large data with batch mode

You can post multiple records with a single request by packing data into a JSON/MessagePack array.

# Send multiple events as a JSON array
$ curl -X POST -d "json=[{"foo":"bar"},{"abc":"def"},{"xyz":"123"}]" \
  http://localhost:9880/app.log

This significantly improves the throughput since it reduces the number of HTTP requests. Here is a simple bechmark result on MacBook Pro with ruby 2.3:

json msgpack msgpack array(10 items)
2100 events/sec 2400 events/sec 10000 events/sec

Tested configuration and ruby script is here.

Multi-process environment

If you use this plugin under multi-process environment, port will be shared.

<system>
  workers 3
</system>
<source>
  @type http
  port 9880
</source>

With this configuration, 3 workers share 9880 port. No need additional port. Incoming data will be routed to 3 workers automatically.

Troubleshooting

Why in_http removes ‘+’ from my log?

This is HTTP spec, not fluentd problem. You need to encode your payload properly or use multipart request. Here is ruby example:

# OK
URI.encode_www_form({json: {"message" => "foo+bar"}.to_json})

# NG
"json=#{"message" => "foo+bar"}.to_json}"

curl command example:

# OK
curl -X POST -H 'Content-Type: multipart/form-data' -F 'json={"message":"foo+bar"}' http://localhost:9880/app.log

# NG
curl -X POST -F 'json={"message":"foo+bar"}' http://localhost:9880/app.log

Learn More

Last updated: 2018-02-24 11:55:09 +0000

Versions | v1.0 (td-agent3) | v0.12 (td-agent2)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.