http
The in_http
Input plugin allows you to send events through HTTP requests. Using this plugin, you can trivially launch a REST endpoint to gather data.
Configuration
Here is a sample configuration:
For the full list of the configurable options, see the Parameters section.
Basic Usage
By default, the data format depends on the Content-Type
. In summary, you can send the following format.
json
ndjson
msgpack
Here is a simple example to post a record using curl
, which uses the default Content-Type application/x-www-form-urlencoded
.
Example: post JSON data with the tag "app.log":
Example: post NDJSON data with the tag "app.log":
Example: post MessagePack data with the tag "app.log":
Some Media Types other than application/x-www-form-urlencoded
support specific formats. If those Media Types are specified with Content-Type:
, the prefix such as json=
is not necessary for posting data.
Example: Post JSON data with Content-Type: application/json
:
For more details regarding the message body syntax and Content-Type
see How to use HTTP Content-Type Header
By default, timestamps are assigned to each record on arrival. You can override the timestamp using the time
parameter:
Here is another example in JavaScript:
For more advanced usage, please read the Tips and Tricks section.
Parameters
See Common Parameters.
@type
(required)
@type
(required)The value must be http
.
port
port
type | default | version |
---|---|---|
integer | 9880 | 0.14.0 |
The port to listen to.
bind
bind
type | default | version |
---|---|---|
string | 0.0.0.0 (all addresses) | 0.14.0 |
The bind address to listen to.
body_size_limit
body_size_limit
type | default | version |
---|---|---|
size | 32MB | 0.14.0 |
The size limit of the POSTed element.
keepalive_timeout
keepalive_timeout
type | default | version |
---|---|---|
size | 10 (seconds) | 0.14.0 |
The timeout limit for keeping the connection alive.
add_http_headers
add_http_headers
type | default | version |
---|---|---|
bool | false | 0.14.0 |
Adds HTTP_
prefix headers to the record.
add_remote_addr
add_remote_addr
type | default | version |
---|---|---|
bool | false | 0.14.0 |
Adds REMOTE_ADDR
field to the record. The value of REMOTE_ADDR
is the client's address.
If your system set multiple X-Forwarded-For
headers in the request, in_http
uses the first one. For example:
If the above multiple headers are sent, the value of REMOTE_ADDR
will be host1
.
cors_allow_origins
cors_allow_origins
type | default | version |
---|---|---|
array | nil(disabled) | 0.14.0 |
Whitelist domains for CORS.
If you set ["domain1", "domain2"]
to cors_allow_origins
, in_http
returns 403
to access from other domains. Since Fluentd v1.2.6, you can use a wildcard character *
to allow requests from any origins.
Example:
cors_allow_credentials
cors_allow_credentials
type | default | version |
---|---|---|
bool | false | 1.14.0 |
Add Access-Control-Allow-Credentials
header. It's needed when a request's credentials mode is include
. An example of use case is using Beacon API, its request mode is always include
.
respond_with_empty_img
respond_with_empty_img
type | default | version |
---|---|---|
bool | false | 0.12.0 |
Responds with an empty GIF image of 1x1 pixel (rather than an empty string).
use_204_response
use_204_response
type | default | version |
---|---|---|
bool | false | v1.8.0 |
Respond status code with 204. This option will be deprecated at v2 because fluentd v2 will respond 204 as default.
<transport>
Section
<transport>
Sectiontype | default | available values | version |
---|---|---|---|
enum | tcp | tcp, tls | 1.5.0 |
This section is for setting TLS transport or some general transport configurations.
General configuration
linger_timeout
type | default | available transport type | version |
---|---|---|---|
integer | 0 | tcp, tls | 1.14.6 |
The timeout (seconds) to set SO_LINGER
.
The default value 0
is to send RST rather than FIN to avoid lots of connections sitting in TIME_WAIT on closing on non-Windows.
You can set positive value to send FIN on closing on non-Windows.
(On Windows, Fluentd sends FIN when linger_timeout
is 0
too).
TLS configuration
See How to Enable TLS Encryption section for how to use and see Configuration Example for all supported parameters.
Without <transport tls>
, in_http
uses HTTP.
<parse>
directive
<parse>
directiveUse the parser plugin to parse the incoming data. See also Handle other formats using parser plugins section.
format
(deprecated)
format
(deprecated)Deprecated parameter. Use <parse>
directive instead.
Tips and Tricks
How to send data in MessagePack format?
You can post data in MessagePack format by adding the msgpack=
prefix:
How to use HTTP Content-Type header?
in_http
plugin recognizes HTTP Content-Type
header in the incoming requests.
If you use the default <parse>
setting, the data format depends on the Content-Type
. (If you set the <parse>
directive to use a specific Parser, the Content-Type
is not used).
By default curl
uses -H "Content-Type: application/x-www-form-urlencoded"
, which allows the use of the prefix json=
, ndjson=
, and msgpack=
as seen on the previous examples.
On the other hand, some Media Types other than application/x-www-form-urlencoded
support specific formats. If those Media Types are specified with Content-Type:
, the prefix such as json=
is not necessary for posting data.
Here is the list of supported Media Types:
Media Types | data format | version |
---|---|---|
| JSON | - |
| JSON | 1.17.0 |
| MessagePack | - |
| NDJSON | 1.14.5 |
Examples:
Also, you can use multipart/form-data
. For more details about multipart/form-data
, please see Why in_http
removes '+' from my log.
Handle Other Formats using Parser Plugins
You can handle various input formats by using the <parse>
directive. For example, add the following settings to the configuration file:
Now you can post custom-format records like this:
Many other formats (e.g. csv
/syslog
/nginx
) are also supported. For the full list of supported formats, see Parser Plugin Overview.
NOTE: Some parser plugins do not support the batch mode. So, if you want to use bulk insertion for handling a large data set, please consider keeping the default JSON (or MessagePack) format or write batch mode supported parser (return array object).
Enhance Performance
Handle Large Data with Batch Mode
You can post multiple records with a single request by packing data into a JSON/MessagePack array:
This significantly improves the throughput since it reduces the number of HTTP requests. Here is a simple benchmark on MacBook Pro with Ruby 2.3:
json | msgpack | msgpack array(10 items) |
---|---|---|
2100 events/sec | 2400 events/sec | 10000 events/sec |
Tested configuration and Ruby script are here.
Use Compression to Reduce Bandwidth Overhead
Since v1.2.3, Fluentd can handle gzip-compressed payloads. To enable this feature, you need to add the Content-Encoding
header to your requests.
You do not need any configuration to enable this feature.
Multi-process Environment
If you use this plugin under the multi-process environment, the port will be shared.
With this configuration, three (3) workers share 9880 port. No need for an additional port. Incoming data will be routed to three (3) workers automatically.
Troubleshooting
Why in_http
splits the messages from my log when using json=
syntax ?
in_http
splits the messages from my log when using json=
syntax ?This happens when using the content type application/x-www-form-urlencoded
. It is a limitation of the HTTP spec, you either need to encode the message or you can use the content type application/json
.
For example:
Why in_http
removes '+' from my log?
in_http
removes '+' from my log?This is HTTP spec, not fluentd problem. You need to encode your payload properly or use multipart request. Here is a Ruby example:
curl
command example:
Learn More
Tips
How to Enable TLS Encryption?
Since v1.5.0, in_http
support TLS transport. Here is a configuration example with HTTPS client:
https client
How to Enable TLS Mutual Authentication?
Fluentd supports TLS mutual authentication (i.e. client certificate auth). If you want to use this feature, please set the client_cert_auth
and ca_path
options like this:
When this feature is enabled, Fluentd will check all the incoming requests for a client certificate signed by the trusted CA. Requests with an invalid client certificate will fail.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last updated