Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Example Configuration
  • Example Usage
  • Plugin Helpers
  • Parameters
  • @type (required)
  • tag (required)
  • port
  • bind
  • protocol_type
  • <transport> Section
  • message_length_limit
  • frame_type
  • format
  • <parse> Directive
  • message_format
  • with_priority
  • emit_unmatched_lines
  • resolve_hostname
  • send_keepalive_packet
  • source_hostname_key
  • source_address_key
  • severity_key
  • priority_key
  • facility_key
  • TCP Protocol and Message Delimiter
  • Tips
  • How to Enable TLS Encryption
  • Multi-process Environment
  • FAQ
  • Our system sends RFC3164/RFC5424 message but parse failure happens
  • Learn More

Was this helpful?

  1. Input Plugins

syslog

PrevioushttpNextexec

Last updated 5 months ago

Was this helpful?

The in_syslog Input plugin enables Fluentd to retrieve records via the syslog protocol on UDP or TCP.

It is included in Fluentd's core.

Example Configuration

<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  tag system
</source>

This tells Fluentd to create a socket listening on port 5140. You need to set up your syslog daemon to send messages to the socket. For example, if you're using rsyslogd, add the following lines to /etc/rsyslog.conf:

# Send log messages to Fluentd
*.* @127.0.0.1:5140

Example Usage

tag = "#{@tag}.#{facility}.#{priority}"
time = 1353436518,
record = {
  "host": "host",
  "ident": "ident",
  "pid": "12345",
  "message": "text"
}

If you want to keep facility and priority in the record, set related parameters.

Plugin Helpers

Parameters

@type (required)

The value must be syslog.

tag (required)

port

type
default
version

integer

5140

0.14.0

The port to listen to.

bind

type
default
version

string

0.0.0.0 (all addresses)

0.14.0

The bind address to listen to.

protocol_type

type
default
available values
version

enum

udp

udp/tcp

0.14.0

The transport protocol used to receive logs. udp and tcp are supported.

This parameter is deprecated since v1.5. Use <transport> instead.

<transport> Section

type
default
available values
version

enum

udp

udp/tcp/tls

1.5.0

The protocol of the syslog transport.

<source>
  @type syslog
  tag system
  <transport tcp>
  </transport>
  # other parameters
</source>

message_length_limit

type
default
version

size

2048

0.14.2

The maximum length of a syslog message in bytes. If you send a larger message, change this parameter.

frame_type

type
default
available values
version

enum

traditional

traditional/octet_count

1.3.0

Specifies the framing type in TCP protocol.

  • traditional

Messages are delimited by newline():

<6>Sep 10 00:00:00 localhost logger: hello!\n
  • octet_count

Message has the message size prefix to delimit:

43 <6>Sep 10 00:00:00 localhost logger: hello!

format

Deprecated parameter. Use <parse> instead.

<parse> Directive

<source>
  @type syslog
  tag system
  <parse>
    @type FORMAT_PARAMETER
  </parse>
</source>

Your <parse> regexp should not consider the 'priority' prefix of the log. For example, if in_syslog receives the log below:

<1>Feb 20 00:00:00 192.168.0.1 fluentd[11111]: [error] hogehoge

Then, the format parser receives the following log:

Feb 20 00:00:00 192.168.0.1 fluentd[11111]: [error] hogehoge

If the <parse>/@type parameter is missing, the log data is assumed to have the canonical syslog format. It is same with the following configuration:

<parse>
  @type syslog
  with_priority true
</parse>

message_format

type
default
available values
version

enum

rfc3164

rfc3164/rfc5424/auto

0.14.14

This parameter is used inside <parse> directive. The default is rfc3164.

<source>
  @type syslog
  tag system
  <parse>
    message_format rfc5424
  </parse>
</source>

Specifies the protocol format. Supported values are rfc3164, rfc5424 and auto. If your syslog uses rfc5424, use rfc5424 instead. Here is an example of message:

# rfc3164
<6>Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Hello!
# rfc5424
<16>1 2017-02-28T12:00:00.009Z 192.168.0.1 fluentd - - - Hello!

auto is useful when in_syslog receives both rfc3164 and rfc5424 message per source. in_syslog detects message format by using message prefix and parses it.

with_priority

type
default
version

bool

true

0.14.0

This parameter is used inside <parse> directive.

<source>
  @type syslog
  tag system
  <parse>
    with_priority false
  </parse>
</source>

If with_priority is true, then syslog messages are assumed to be prefixed with a priority tag like <3>. This option exists since some syslog daemons output logs without the priority tag preceding the message body.

emit_unmatched_lines

type
default
version

bool

false

1.6.3

Emits unmatched lines when <parse> format is not matched for incoming logs.

Emitted record is {"unmatched_line" : "incoming line"} with ${tag parameter}.unmatched tag.

resolve_hostname

type
default
version

bool

nil

0.14.19

Tries to resolve hostname from IP addresses or not. Cannot set false when source_hostname_key is set.

send_keepalive_packet

type
default
version

bool

false

1.14.0

source_hostname_key

type
default
version

string

nil (no assign)

0.14.0

The field name of the client's hostname. If set, the client's hostname will be set to its key.

source_address_key

type
default
version

string

nil (no assign)

0.14.0

The field name of the client's address. If set, the client's address will be set to its key.

severity_key

type
default
version

string

nil (no assign)

1.7.3

The field name of the severity. If set, the severity will be set to its key.

If you set severity_key severity and got <6> started syslog message, severity field is info.

priority_key

type
default
version

string

nil (no assign)

0.14.10

This parameter is deprecated due to a misleading name. This sets severity, not priority.

This parameter will be removed in fluentd v2. Use severity_key instead.

facility_key

type
default
version

string

nil (no assign)

0.14.10

The field name of the facility. If set, the facility will be set to its key.

If you set facility_key facility and got <6> started syslog message, facility field is kern.

@log_level

The @log_level option allows the user to set different levels of logging for each plugin. The supported log levels are: fatal, error, warn, info, debug, and trace.

TCP Protocol and Message Delimiter

If your syslog uses octet counting mode, set frame_type octet_count in in_syslog configuration. See also frame_type parameter.

Tips

How to Enable TLS Encryption

Since v1.5.0, in_syslog support TLS transport. Here is the configuration example with rsyslog:

  • in_syslog

<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  <transport tls>
    ca_path /etc/pki/ca.pem
    cert_path /etc/pki/cert.pem
    private_key_path /etc/pki/key.pem
    private_key_passphrase PASSPHRASE
  </transport>
  tag system
</source>
  • rsyslog

$DefaultNetstreamDriverCAFile /etc/pki/ca.pem
$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
*.* @@127.0.0.1:5140

Multi-process Environment

If you use this plugin under the multi-process environment, the port will be shared.

<system>
  workers 3
</system>

<source>
  @type syslog
  port 5140
</source>

With this configuration, 3 workers share 5140 port. No need of an additional port. The incoming data will be routed to the three (3) workers automatically.

FAQ

Our system sends RFC3164/RFC5424 message but parse failure happens

First, check your message format follows RFC3164/RFC5424 or not. Some systems say RFC3164/RFC5424 but it sends non-RFC3164/RFC5424 message, e.g. invalid priority, different timestamp, lack/add fields.

If only timestamp is different, configure time_format in <parse> may help.

If other parts are different, the syslog parser cannot parse your message. To resolve the problem, there are several approaches:

  • Use regex parser or write your parser

  • Use in_udp/in_tcp with other parsers

Learn More

The retrieved data is organized as follows. Fluentd's tag is generated by the tag parameter (tag prefix), , and . The record is parsed by the regexp .

See .

The prefix of the tag. The tag itself is generated by the tag prefix, , and .

This section is for setting TLS transport or some general transport configurations. See How to Enable TLS Encryption section for how to use and see and for all supported parameters.

See also .

The format of the log. This option is used to parse non-standard syslog formats using .

If you wish to parse syslog messages of arbitrary formats, or are recommended.

Enables the TCP keepalive for sockets. See for more details.

Please see the for further details.

This plugin assumes for delimiter character between syslog messages in one TCP connection by default. If you use syslog library in your application with <transport tcp>, add to your syslog message. See also .

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

facility level
priority
here
parser
compat_parameters
server
Common Parameters
facility level
priority
RFC 6587
parser plugins
in_tcp
in_udp
logging article
rfc6587
Input Plugin Overview
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
Config: Transport Section
Configuration Example
socket article