The in_http Input plugin allows you to send events through HTTP requests. Using this plugin, you can trivially launch a REST endpoint to gather data.

Configuration

The following snippet shows an example configuration.

<source>
  @type http
  port 9880
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
</source>

For the full list of the configurable options, see the "Parameters" section.

Basic Usage

Here is a simple example to post a record using curl.

# Post a record with the tag "app.log"
$ curl -X POST -d 'json={"foo":"bar"}' http://localhost:9880/app.log

By default, timestamps are assigned to each record on arrival. You can override the timestamp using the time parameter.

# Overwrite the timestamp to 2018-02-16 04:40:37.3137116
$ curl -X POST -d 'json={"foo":"bar"}' \
  http://localhost:9880/test.tag?time=1518756037.3137116

Here is another example in JavaScript.

// Post a record using XMLHttpRequest
var form = new FormData();
form.set('json', JSON.stringify({"foo": "bar"}));
​
var req = new XMLHttpRequest();
req.open('POST', 'http://localhost:9880/debug.log');
req.send(form);

For more advanced usage, please read the "Tips & Tricks" section.

Parameters

​Common Parameters​

@type (required)

The value must be http.

port

The port to listen to.

bind

The bind address to listen to.

body_size_limit

The size limit of the POSTed element.

keepalive_timeout

The timeout limit for keeping the connection alive.

add_http_headers

Add HTTP_ prefix headers to the record.

add_remote_addr

Add REMOTE_ADDR field to the record. The value of REMOTE_ADDR is the client's address.

If your system set multiple X-Forwarded-For headers in the request, in_http uses first one. For example:

X-Forwarded-For: host1, host2
X-Forwarded-For: host3

If send above multiple headers, REMOTE_ADDR value is host1.

cors_allow_origins

White list domains for CORS.

If you set ["domain1", "domain2"] to cors_allow_origins, in_http returns 403 to access from other domains. Since Fluentd v1.2.6, you can use a wildcard character * to allow requests from any origins (see the following example).

<source>
  @type http
  port 9880
  cors_allow_origins ["*"]
</source>

respond_with_empty_img

Respond with an empty gif image of 1x1 pixel (rather than an emtpy string).

<transport> section

This section is for using TLS transport.

<transport tls>
  cert_path /path/to/fluentd.crt
  # other parameters
</transport>

See "How to Enable TLS Encryption" section for how to use and see "Configuration example" in "Server Plugin Helper" article for supported parameters

Without <transport tls>, in_http uses HTTP.

<parse> directive

Use parser plugin to parse incoming data. See also Handle other formats using parser plugins section.

format (deprecated)

Deprecated parameter. Use <parse> directive instead.

Tips & Tricks

How to send data in MessagePack format

You can post data in MessagePack format by adding the msgpack= prefix.

# Send data in msgpack format
$ msgpack=`echo -e "\x81\xa3foo\xa3bar"`
$ curl -X POST -d "msgpack=$msgpack" http://localhost:9880/app.log

How to use HTTP Content-Type header

in_http plugin recognizes HTTP 'Content-Type' header in the incoming requests. For example, you can send a JSON payload without the json= prefix.

$ curl -X POST -d '{"foo":"bar"}' -H 'Content-Type: application/json' \
  http://localhost:9880/app.log

To use MessagePack, set the content type to application/msgpack.

$ msgpack=`echo -e "\x81\xa3foo\xa3bar"`
$ curl -X POST -d "$msgpack" -H 'Content-Type: application/msgpack' \
  http://localhost:9880/app.log

Handle other formats using parser plugins

You can handle various input formats by using the <parse> directive. For example, add the following settings to the configuration file:

<source>
  @type http
  port 9880
  <parse>
    @type regexp
    expression /^(?<field1>\d+):(?<field2>\w+)$/
  </parse>
</source>

Now you can post custom-format records as below:

# This will be parsed into {"field1":"123456","field2":"awesome"}
$ curl -X POST -d '123456:awesome' http://localhost:9880/app.log

Many other formats (e.g. csv/syslog/nginx) are also supported as well. You can find the full list of supported formats in "Parser Plugin Overview".

Note: Some parser plugins do not support the batch mode. So if you want to use bulk insertion for handling a large data set, please consider to keep using the default JSON (or MessagePack) format or write batch mode supported parser(return array object).

Enhance Performance

Handle large data with batch mode

You can post multiple records with a single request by packing data into a JSON/MessagePack array.

# Send multiple events as a JSON array
$ curl -X POST -d 'json=[{"foo":"bar"},{"abc":"def"},{"xyz":"123"}]' \
  http://localhost:9880/app.log

This significantly improves the throughput since it reduces the number of HTTP requests. Here is a simple bechmark result on MacBook Pro with ruby 2.3:

Tested configuration and ruby script is here.

Use compression to reduce bandwidth overhead

Since v1.2.3, Fluentd can handle gzip-compressed payloads. To enable this feature, you need to add the 'Content-Encoding' header to your requests.

# Send gzip-compressed payload
$ echo 'json={"foo":"bar"}' | gzip > json.gz
$ curl --data-binary @json.gz -H "Content-Encoding: gzip" \
  http://localhost:9880/app.log

You do not need any configuration to enable this feature.

Multi-process environment

If you use this plugin under multi-process environment, port will be shared.

<system>
  workers 3
</system>
<source>
  @type http
  port 9880
</source>

With this configuration, 3 workers share 9880 port. No need additional port. Incoming data will be routed to 3 workers automatically.

Troubleshooting

Why in_http removes '+' from my log?

This is HTTP spec, not fluentd problem. You need to encode your payload properly or use multipart request. Here is ruby example:

# OK
URI.encode_www_form({json: {"message" => "foo+bar"}.to_json})
​
# NG
"json=#{"message" => "foo+bar"}.to_json}"

curl command example:

# OK
curl -X POST -H 'Content-Type: multipart/form-data' -F 'json={"message":"foo+bar"}' http://localhost:9880/app.log
​
# NG
curl -X POST -F 'json={"message":"foo+bar"}' http://localhost:9880/app.log

Learn More

• ​Input Plugin Overview​

Tips

How to Enable TLS Encryption

Since v1.5.0, in_http support TLS tranport. Here is configuration example with HTTPS client.

• in_tcp

<source>
  @type http
  bind 0.0.0.0
  <transport tls>
    ca_path /etc/pki/ca.pem
    cert_path /etc/pki/cert.pem
    private_key_path /etc/pki/key.pem
    private_key_passphrase PASSPHRASE
  </transport>
</source>

• https client

require 'net/http'
require 'net/https'
require 'msgpack'
​
record = { 'msgpack' => { 'k' => 'hello', 'k1' => 1234}.to_msgpack }
​
def post(path, params)
  http = Net::HTTP.new('127.0.0.1', 9880)
  http.use_ssl = true
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE
  req = Net::HTTP::Post.new(path, {})
  req.set_form_data(params)
  http.request(req)
end
​
puts post("/test.http?time=#{Time.now.to_i}", record).body

How to Enable TLS Mutual Authentication

Fluentd supports TLS mutual authentication (a.k.a. client certificate auth). If you want to use this feature, please set the client_cert_auth and ca_path options as follows.

<source>
  @type http
  <transport tls>
    ...
    client_cert_auth true
    ca_path /path/to/ca/cert
  </transport>
</source>

When this feature is enabled, Fluentd will check all incoming requests for a client certificate signed by the trusted CA. Requests that don't supply a valid client certificate will fail.

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.