Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Configuration
  • Basic Usage
  • Parameters
  • @type (required)
  • port
  • bind
  • body_size_limit
  • keepalive_timeout
  • add_http_headers
  • add_remote_addr
  • cors_allow_origins
  • cors_allow_credentials
  • respond_with_empty_img
  • use_204_response
  • <transport> Section
  • <parse> directive
  • format (deprecated)
  • Tips and Tricks
  • How to send data in MessagePack format?
  • How to use HTTP Content-Type header?
  • Handle Other Formats using Parser Plugins
  • Enhance Performance
  • Handle Large Data with Batch Mode
  • Use Compression to Reduce Bandwidth Overhead
  • Multi-process Environment
  • Troubleshooting
  • Why in_http splits the messages from my log when using json= syntax ?
  • Why in_http removes '+' from my log?
  • Learn More
  • Tips
  • How to Enable TLS Encryption?
  • How to Enable TLS Mutual Authentication?

Was this helpful?

  1. Input Plugins

http

PreviousunixNextsyslog

Last updated 8 months ago

Was this helpful?

The in_http Input plugin allows you to send events through HTTP requests. Using this plugin, you can trivially launch a REST endpoint to gather data.

Configuration

Here is a sample configuration:

<source>
  @type http
  port 9880
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
</source>

Basic Usage

By default, the data format depends on the Content-Type. In summary, you can send the following format.

  • json

  • ndjson

  • msgpack

Here is a simple example to post a record using curl, which uses the default Content-Type application/x-www-form-urlencoded.

Example: post JSON data with the tag "app.log":

curl -X POST -d 'json={"foo":"bar"}' http://localhost:9880/app.log

Example: post NDJSON data with the tag "app.log":

ndjson=`echo -e 'ndjson={"k1":"v1"}\n{"k2":"v2"}\n'`
curl -X POST -d "$ndjson" http://localhost:9880/app.log

Example: post MessagePack data with the tag "app.log":

msgpack=`echo -e "msgpack=\x81\xa3foo\xa3bar"`
curl -X POST -d "$msgpack" http://localhost:9880/app.log

Some Media Types other than application/x-www-form-urlencoded support specific formats. If those Media Types are specified with Content-Type: , the prefix such as json= is not necessary for posting data.

Example: Post JSON data with Content-Type: application/json:

curl -X POST -d '{"foo":"bar"}' -H 'Content-Type: application/json' \
  http://localhost:9880/app.log

By default, timestamps are assigned to each record on arrival. You can override the timestamp using the time parameter:

# Overwrite the timestamp to 2018-02-16 04:40:37.3137116
$ curl -X POST -d 'json={"foo":"bar"}' \
  http://localhost:9880/test.tag?time=1518756037.3137116

Here is another example in JavaScript:

// Post a record using XMLHttpRequest
var form = new FormData();
form.set('json', JSON.stringify({"foo": "bar"}));

var req = new XMLHttpRequest();
req.open('POST', 'http://localhost:9880/debug.log');
req.send(form);

Parameters

@type (required)

The value must be http.

port

type
default
version

integer

9880

0.14.0

The port to listen to.

bind

type
default
version

string

0.0.0.0 (all addresses)

0.14.0

The bind address to listen to.

body_size_limit

type
default
version

size

32MB

0.14.0

The size limit of the POSTed element.

keepalive_timeout

type
default
version

size

10 (seconds)

0.14.0

The timeout limit for keeping the connection alive.

add_http_headers

type
default
version

bool

false

0.14.0

Adds HTTP_ prefix headers to the record.

add_remote_addr

type
default
version

bool

false

0.14.0

Adds REMOTE_ADDR field to the record. The value of REMOTE_ADDR is the client's address.

If your system set multiple X-Forwarded-For headers in the request, in_http uses the first one. For example:

X-Forwarded-For: host1, host2
X-Forwarded-For: host3

If the above multiple headers are sent, the value of REMOTE_ADDR will be host1.

cors_allow_origins

type
default
version

array

nil(disabled)

0.14.0

Whitelist domains for CORS.

If you set ["domain1", "domain2"] to cors_allow_origins, in_http returns 403 to access from other domains. Since Fluentd v1.2.6, you can use a wildcard character * to allow requests from any origins.

Example:

<source>
  @type http
  port 9880
  cors_allow_origins ["*"]
</source>

cors_allow_credentials

type
default
version

bool

false

1.14.0

respond_with_empty_img

type
default
version

bool

false

0.12.0

Responds with an empty GIF image of 1x1 pixel (rather than an empty string).

use_204_response

type
default
version

bool

false

v1.8.0

Respond status code with 204. This option will be deprecated at v2 because fluentd v2 will respond 204 as default.

<transport> Section

type
default
available values
version

enum

tcp

tcp, tls

1.5.0

This section is for setting TLS transport or some general transport configurations.

General configuration

linger_timeout

type
default
available transport type
version

integer

0

tcp, tls

1.14.6

The timeout (seconds) to set SO_LINGER.

The default value 0 is to send RST rather than FIN to avoid lots of connections sitting in TIME_WAIT on closing on non-Windows.

You can set positive value to send FIN on closing on non-Windows.

On Windows, Fluentd sends FIN without depending on this setting.

<transport tcp>
  linger_timeout 1
</transport>

TLS configuration

<transport tls>
  cert_path /path/to/fluentd.crt
  # other parameters
</transport>

Without <transport tls>, in_http uses HTTP.

<parse> directive

format (deprecated)

Deprecated parameter. Use <parse> directive instead.

Tips and Tricks

How to send data in MessagePack format?

You can post data in MessagePack format by adding the msgpack= prefix:

# Send data in msgpack format
$ msgpack=`echo -e "\x81\xa3foo\xa3bar"`
$ curl -X POST -d "msgpack=$msgpack" http://localhost:9880/app.log

How to use HTTP Content-Type header?

in_http plugin recognizes HTTP Content-Type header in the incoming requests.

If you use the default <parse> setting, the data format depends on the Content-Type. (If you set the <parse> directive to use a specific Parser, the Content-Type is not used).

By default curl uses -H "Content-Type: application/x-www-form-urlencoded", which allows the use of the prefix json=, ndjson=, and msgpack= as seen on the previous examples.

On the other hand, some Media Types other than application/x-www-form-urlencoded support specific formats. If those Media Types are specified with Content-Type: , the prefix such as json= is not necessary for posting data.

Here is the list of supported Media Types:

Media Types
data format
version

application/json

JSON

-

application/csp-report

JSON

1.17.0

application/msgpack

MessagePack

-

application/x-ndjson

NDJSON

1.14.5

Examples:

curl -X POST -d '{"foo":"bar"}' -H 'Content-Type: application/json' \
  http://localhost:9880/app.log
msgpack=`echo -e "\x81\xa3foo\xa3bar"`
curl -X POST -d "$msgpack" -H 'Content-Type: application/msgpack' \
  http://localhost:9880/app.log

Handle Other Formats using Parser Plugins

You can handle various input formats by using the <parse> directive. For example, add the following settings to the configuration file:

<source>
  @type http
  port 9880
  <parse>
    @type regexp
    expression /^(?<field1>\d+):(?<field2>\w+)$/
  </parse>
</source>

Now you can post custom-format records like this:

# This will be parsed into {"field1":"123456","field2":"awesome"}
$ curl -X POST -d '123456:awesome' http://localhost:9880/app.log

Enhance Performance

Handle Large Data with Batch Mode

You can post multiple records with a single request by packing data into a JSON/MessagePack array:

# Send multiple events as a JSON array
$ curl -X POST -d 'json=[{"foo":"bar"},{"abc":"def"},{"xyz":"123"}]' \
  http://localhost:9880/app.log

This significantly improves the throughput since it reduces the number of HTTP requests. Here is a simple benchmark on MacBook Pro with Ruby 2.3:

json
msgpack
msgpack array(10 items)

2100 events/sec

2400 events/sec

10000 events/sec

Use Compression to Reduce Bandwidth Overhead

Since v1.2.3, Fluentd can handle gzip-compressed payloads. To enable this feature, you need to add the Content-Encoding header to your requests.

# Send gzip-compressed payload
$ echo 'json={"foo":"bar"}' | gzip > json.gz
$ curl --data-binary @json.gz -H "Content-Encoding: gzip" \
  http://localhost:9880/app.log

You do not need any configuration to enable this feature.

Multi-process Environment

If you use this plugin under the multi-process environment, the port will be shared.

<system>
  workers 3
</system>
<source>
  @type http
  port 9880
</source>

With this configuration, three (3) workers share 9880 port. No need for an additional port. Incoming data will be routed to three (3) workers automatically.

Troubleshooting

Why in_http splits the messages from my log when using json= syntax ?

This happens when using the content type application/x-www-form-urlencoded. It is a limitation of the HTTP spec, you either need to encode the message or you can use the content type application/json.

For example:


# bad
curl -X POST --location "http://localhost:9880/app.loge" \
        -d "json=[{\"log\":\"message ; remaining message\"}]"
# good
curl -X POST --location "http://localhost:9880/app.log" \
        -H "Content-Type: application/json" \
        -d "{\"log\":\"message ; remaining message\"}"

Why in_http removes '+' from my log?

This is HTTP spec, not fluentd problem. You need to encode your payload properly or use multipart request. Here is a Ruby example:

# Good
URI.encode_www_form({json: {"message" => "foo+bar"}.to_json})

# Bad
"json=#{"message" => "foo+bar"}.to_json}"

curl command example:

# Good
curl -X POST -H 'Content-Type: multipart/form-data' -F 'json={"message":"foo+bar"}' http://localhost:9880/app.log

# Bad
curl -X POST -F 'json={"message":"foo+bar"}' http://localhost:9880/app.log

Learn More

Tips

How to Enable TLS Encryption?

Since v1.5.0, in_http support TLS transport. Here is a configuration example with HTTPS client:

<source>
  @type http
  bind 0.0.0.0
  <transport tls>
    ca_path /etc/pki/ca.pem
    cert_path /etc/pki/cert.pem
    private_key_path /etc/pki/key.pem
    private_key_passphrase PASSPHRASE
  </transport>
</source>
  • https client

require 'net/http'
require 'net/https'
require 'msgpack'

record = { 'msgpack' => { 'k' => 'hello', 'k1' => 1234}.to_msgpack }

def post(path, params)
  http = Net::HTTP.new('127.0.0.1', 9880)
  http.use_ssl = true
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE
  req = Net::HTTP::Post.new(path, {})
  req.set_form_data(params)
  http.request(req)
end

puts post("/test.http?time=#{Time.now.to_i}", record).body

How to Enable TLS Mutual Authentication?

<source>
  @type http
  <transport tls>
    ...
    client_cert_auth true
    ca_path /path/to/ca/cert
  </transport>
</source>

When this feature is enabled, Fluentd will check all the incoming requests for a client certificate signed by the trusted CA. Requests with an invalid client certificate will fail.

For the full list of the configurable options, see the section.

For more details regarding the message body syntax and Content-Type see

For more advanced usage, please read the section.

See .

Add header. It's needed when a request's credentials mode is include. An example of use case is using , its request mode is always include.

See How to Enable TLS Encryption section for how to use and see for all supported parameters.

Use the parser plugin to parse the incoming data. See also section.

Also, you can use multipart/form-data. For more details about multipart/form-data, please see .

Many other formats (e.g. csv/syslog/nginx) are also supported. For the full list of supported formats, see .

NOTE: Some parser plugins do not support the . So, if you want to use bulk insertion for handling a large data set, please consider keeping the default JSON (or MessagePack) format or write batch mode supported parser (return array object).

Tested configuration and Ruby script are .

Fluentd supports (i.e. client certificate auth). If you want to use this feature, please set the client_cert_auth and ca_path options like this:

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

Common Parameters
Access-Control-Allow-Credentials
Beacon API
Parser Plugin Overview
here
Input Plugin Overview
TLS mutual authentication
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
Parameters
How to use HTTP Content-Type Header
Tips and Tricks
Handle other formats using parser plugins
Why in_http removes '+' from my log
batch mode
Configuration Example