Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • How to launch Fluentd with source-only mode
  • Temporary file buffer
  • Recovery
  • Mechanism

Was this helpful?

  1. Deployment

Source Only Mode

PreviousCommand Line OptionNextZero-downtime restart

Last updated 3 months ago

Was this helpful?

Since v1.18.0, Fluentd can launch with source-only mode. (Not supported on Windows)

  • In this mode, only input plugins run.

  • During this mode, the events are stored in a temporary file buffer.

  • Sending SIGWINCH to the supervisor cancels this mode.

    • Then, all plugins start to run, and the temporary file buffer starts to load.

How to launch Fluentd with source-only mode

You can launch Fluentd with source-only mode in the following ways.

  • Command Line Option - --with-source-only

Temporary file buffer

During source-only mode, the ingested events are stored in a temporary file buffer. After SIGWINCH is sent to the supervisor and this mode is canceled, this buffer begins to load.

By default, the file buffer path is as follows.

  • /tmp/fluent/source-only-buffer/{Unique ID for the Fluentd instance}

You can confirm it in the following log output at startup.

[info]: #0 with-source-only: the emitted data will be stored in the buffer files under
/tmp/fluent/source-only-buffer/bbd9006d-bc41-418b-b346-f80888641dda. You can send SIGWINCH to
the supervisor process to cancel with-source-only mode and process data.

This file buffer is the buffer of out_buffer. It works with the default settings of out_buffer except for the following points:

  • path is automatically determined by default.

  • overflow_action is drop_oldest_chunk by default.

If needed, you can configure some options for the buffer in System Configuration. Please see the following for details.

Recovery

If Fluentd stops with the temporary buffer remained, you need to recover the buffer to launch Fluentd with source-only mode again.

Note that a different path will be used each time unless you configure the temporary buffer path explicitly. In this case, you can recover the buffer as follows.

  1. Configure the remaining buffer path explicitly.

  2. Start Fluentd with source-only mode again.

  3. Send SIGWINCH to the supervisor to load the buffer.

If this recovery is necessary, i.e., Fluentd stops with the temporary buffer remained, the following warning log will be displayed. You can confirm the path to configure by this log.

[warn]: #0 some buffer files remain in /tmp/fluent/source-only-buffer/bbd9006d-bc41-418b-b346-f80888641dda.
Please consider recovering or saving the buffer files in the directory. To recover them, you can set
the buffer path manually to system config and retry, i.e., restart Fluentd with with-source-only mode
and send SIGWINCH again. Config Example:
<system>
  <source_only_buffer>
    path /tmp/fluent/source-only-buffer/bbd9006d-bc41-418b-b346-f80888641dda
  </source_only_buffer>
</system>

Mechanism

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.

.

System Configuration - with_source_only
System Configuration - source_only_buffer section
source-only-mode mechanism