Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Prerequisites
  • Install plugin
  • Configuration
  • Input Settings
  • Output Settings
  • Test the Settings
  • Learn More

Was this helpful?

  1. How-to Guides

Send Apache Logs to Minio

PreviousSend Apache Logs to S3NextSend Apache Logs to Mongodb

Last updated 3 months ago

Was this helpful?

This article explains how to use Fluentd to aggregate and transport Apache logs to a server.

Prerequisites

The following software/services are required to be set up correctly:

  • (S3 Compatible Storage)

  • (with the Combined Log Format)

You can install Fluentd via major packaging systems.

Install plugin

If (fluent-plugin-s3) is not installed yet, please install it manually.

See section how to install fluent-plugin-s3 on your environment.

If you use fluent-package, out_s3 (fluent-plugin-s3) is bundled by default.

Configuration

Input Settings

In this example, we use the access log file as an input source, so save the following <source> settings to /etc/fluent/fluentd.conf:

<source>
  @type tail
  format apache2
  path /var/log/apache2/access.log
  pos_file /var/log/fluent/apache.access.log.pos
  tag minio.apache.access
</source>

NOTE: If you are using the standalone version of Fluentd, use /etc/fluent/fluent.conf instead.

Before proceeding, please confirm that the access log file has proper file permission. If the log file is not readable by the fluent-package/fluentd, the rest of this article will not work.

Output Settings

<match minio.apache.**>
  @type s3
  aws_key_id ACCESS_KEY         # The access key for Minio
  aws_sec_key SECRET_KEY        # The secret key for Minio
  s3_bucket BUCKET_NAME         # The bucket to store the log data
  s3_endpoint ENDPOINT          # The endpoint URL (like "http://localhost:9000/")
  s3_region us-east-1           # See the region settings of your Minio server
  path logs/                    # This prefix is added to each file
  time_slice_format %Y%m%d%H%M  # This timestamp is added to each file name

  <buffer time>
    @type file
    path /var/log/fluent/s3
    timekey 60m                 # Flush the accumulated chunks every hour
    timekey_wait 1m             # Wait for 60 seconds before flushing
    timekey_use_utc true        # Use this option if you prefer UTC timestamps
    chunk_limit_size 256m       # The maximum size of each chunk
  </buffer>
</match>

After adding the settings to the conf file, please restart the Fluentd daemon.

Test the Settings

Use curl to generate some log data for testing:

$ curl http://localhost/

Or you can use the Apache Bench for the bulk request generation:

$ ab -n 100 -c 10 http://localhost/

Wait until the data gets flushed from the buffer (you can adjust the flush interval using the timekey and timekey_wait options above). Then you will see the aggregated log data on Minio:

Learn More

Now let's add settings for storing the incoming data in your Minio server. Since Minio is compatible with Amazon Simple Storage Service (S3), we can use the plugin to connect to the server.

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

Minio
Fluentd
Minio
Apache
Installation
out_s3
Plugin Management
out_s3
Fluentd Architecture
Amazon S3 Output plugin
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
Minio