Send Apache Logs to Minio

Search…
This article explains how to use Fluentd to aggregate and transport Apache logs to a Minio server.
Prerequisites
The following services are required to be set up correctly:
​Fluentd​
​Minio​
​Apache​
Also, if you have installed Fluentd through RubyGems (without td-agent), please install the out_s3 plugin manually:
1
$ sudo fluent-gem install fluent-plugin-s3
Copied!
Configuration
Input Settings
In this example, we use the access log file as an input source, so save the following <source> settings to /etc/td-agent/td-agent.conf:
1
<source>
2
  @type tail
3
  format apache2
4
  path /var/log/apache2/access.log
5
  pos_file /var/log/td-agent/apache.access.log.pos
6
  tag minio.apache.access
7
</source>
Copied!
NOTE: If you are using the standalone version of Fluentd, use /etc/fluent/fluent.conf instead.
Before proceeding, please confirm that the access log file has proper file permission. If the log file is not readable by the td-agent/fluentd, the rest of this article will not work.
Output Settings
Now let's add settings for storing the incoming data in your Minio server. Since Minio is compatible with Amazon Simple Storage Service (S3), we can use the out_s3 plugin to connect to the server.
1
<match minio.apache.**>
2
  @type s3
3
  aws_key_id ACCESS_KEY         # The access key for Minio
4
  aws_sec_key SECRET_KEY        # The secret key for Minio
5
  s3_bucket BUCKET_NAME         # The bucket to store the log data
6
  s3_endpoint ENDPOINT          # The endpoint URL (like "http://localhost:9000/")
7
  s3_region us-east-1           # See the region settings of your Minio server
8
  path logs/                    # This prefix is added to each file
9
  force_path_style true         # This prevents AWS SDK from breaking endpoint URL
10
  time_slice_format %Y%m%d%H%M  # This timestamp is added to each file name
11
​
12
  <buffer time>
13
    @type file
14
    path /var/log/td-agent/s3
15
    timekey 60m                 # Flush the accumulated chunks every hour
16
    timekey_wait 1m             # Wait for 60 seconds before flushing
17
    timekey_use_utc true        # Use this option if you prefer UTC timestamps
18
    chunk_limit_size 256m       # The maximum size of each chunk
19
  </buffer>
20
</match>
Copied!
After adding the settings to the conf file, please restart the Fluentd daemon.
Test the Settings
Use curl to generate some log data for testing:
1
$ curl http://localhost/
Copied!
Or you can use the Apache Bench for the bulk request generation:
1
$ ab -n 100 -c 10 http://localhost/
Copied!
Wait until the data gets flushed from the buffer (you can adjust the flush interval using the timekey and timekey_wait options above). Then you will see the aggregated log data on Minio:
Minio
Learn More
​Fluentd Architecture​
​Amazon S3 Output plugin​
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Send Apache Logs to S3
Send Apache Logs to Mongodb
Last modified 4mo ago
Copy link
Contents