Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Which version of Ruby does fluentd support?
  • What is the difference between v1 or v0.14?
  • Operations
  • I have a weird timestamp value, what happened?
  • I installed td-agent/fluent-package and want to add custom plugins. How do I do it?
  • How can I match (send) an event to multiple outputs?
  • How can I use environment variables to configure parameters dynamically?
  • Fluentd raises an error for host:port. Why?
  • fluentd raises tzinfo conflict error after installed plugins
  • I got no patterns matched in the log, why?
  • File buffer does not work properly, why?
  • I got encoding error inside the plugin. How to fix it?
  • Fluentd warns Oj is not installed, and falling back to Yajl for json parser.
  • Plugin Development
  • How do I develop a custom plugin?
  • HOWTOs
  • How can I parse <my complex text log>?

Was this helpful?

  1. Overview

FAQ

PreviousSupportNextLogo

Last updated 9 months ago

Was this helpful?

Which version of Ruby does fluentd support?

Latest fluentd works on Ruby 2.7 or later.

Though the minimum required version is Ruby 2.7 or later, we recommend using fluentd with more newer stable version.

What is the difference between v1 or v0.14?

No difference. v1 is built on top of v0.14. Use v1 for a newer installation. We use v1 or v1.x on our document.

Operations

I have a weird timestamp value, what happened?

The timestamps of Fluentd and its logger libraries depend on your system's clock. It is highly recommended that you set up NTP on your nodes so that your clocks remain synced with the correct clocks.

I installed td-agent/fluent-package and want to add custom plugins. How do I do it?

Use the bundled gem system. See for more information.

How can I match (send) an event to multiple outputs?

You can use the copy to send the same event to multiple output destinations.

How can I use environment variables to configure parameters dynamically?

Use "#{ENV['YOUR_ENV_VARIABLE']}". For example:

some_field "#{ENV['FOO_HOME']}"

Note that it must be double quotes, not single quotes.

Fluentd raises an error for host:port. Why?

There are several reasons:

  • If you get Address already in use error, another process has already

    used host:port. Check port conflict between processes/plugins.

  • If you get Permission denied error, you likely tried to use a well-known

    port without permission. Search well-known ports for how to use well-known ports.

    Use capabilities or something.

If you get other errors, Google it.

fluentd raises tzinfo conflict error after installed plugins

Fluentd supports tzinfo v1.1 or later and recent td-agent / fluent-package / official images install tzinfo v2 by default. The problem is several plugins depend on ActiveSupport and ActiveSupport doesn't support tzinfo v2. To resolve this problem, there are 2 approaches.

  • Uninstall tzinfo v2 and install tzinfo v1.1 manually

  • Update plugin to remove ActiveSupport dependency. ActiveSupport is mainly for Ruby on Rails,

    so using ActiveSupport for several convenient methods is overengineering.

Former is easier approach.

I got no patterns matched in the log, why?

This means that the event is emitted but there is no <match> directive for it. For example, if you emit the event with foo.bar tag, you need to define <match> for foo.bar tag like <match foo.**>.

File buffer does not work properly, why?

I got encoding error inside the plugin. How to fix it?

You may hit "\xC3" from ASCII-8BIT to UTF-8 like UndefinedConversionError in the plugin. This error happens when string encoding is set to ASCII-8BIT but the actual content is UTF-8. Fluentd and all its plugins treat the logs as ASCII-8BIT by default but some libraries assume that the log encoding is UTF-8. This is why this error occurs.

There are several approaches to avoid this problem:

  • Set encoding correctly:

    • The tail input plugin has encoding related parameters to change the

      log encoding.

    • Use record_modifier filter plugin to change the encoding. See

  • Use yajl(Yajl.load/Yajl.dump) instead of json when error happens

    inside JSON.parse/JSON.dump/to_json.

Fluentd warns Oj is not installed, and falling back to Yajl for json parser.

If you are using Alpine Linux, you need to install ruby-bigdecimal to use Oj as the JSON parser. Please Execute the following command to see if the warning persists:

# apk add --update ruby-bigdecimal

Plugin Development

How do I develop a custom plugin?

HOWTOs

How can I parse <my complex text log>?

See also: or

file buffer has limitations. Check .

.

Please refer to the .

If you are willing to write Regexp, or is a great tool to verify your Regexps.

If you do NOT want to write any Regexp, look at .

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under

this section
output plugin
Lifecycle of a Fluentd event
Config File
fluent-plugin-record-modifier README
Plugin Development Guide
the Grok parser
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
the Apache License 2.0.
Fluentular
buf_file article
fluentd-ui's in_tail editor