Collect Glusterfs Logs

This article shows how to use Fluentd to collect GlusterFS logs for analysis (search, analytics, troubleshooting, etc.)
Background
​GlusterFS is an open source, distributed file system commercially supported by Red Hat, Inc. Each node in GlusterFS generates its own logs, and it's sometimes convenient to have these logs collected in a central location for analysis (e.g., When one GlusterFS node went down, what was happening on other nodes?).
​Fluentd is an open source data collector for high-volume data streams. It's a great fit for monitoring GlusterFS clusters because:
1.
Fluentd supports GlusterFS logs as a data source.
2.
Fluentd supports various output systems (e.g., Elasticsearch,
MongoDB, Treasure Data, etc.) that can help GlusterFS users analyze
the logs.
The rest of this article explains how to set up Fluentd with GlusterFS. For this example, we chose Elasticsearch as the backend system.
Setting up Fluentd on GlusterFS Nodes
Step 1: Installing Fluentd
First, we'll install Fluentd using the following command:
1
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
Copied!
Next, we'll install the Fluentd plugin for GlusterFS:
1
$ sudo /usr/sbin/td-agent-gem install fluent-plugin-glusterfs
2
Fetching: fluent-plugin-glusterfs-1.0.0.gem (100%)
3
Successfully installed fluent-plugin-glusterfs-1.0.0
4
1 gem installed
5
Installing ri documentation for fluent-plugin-glusterfs-1.0.0...
6
Installing RDoc documentation for fluent-plugin-glusterfs-1.0.0...
Copied!
Step 2: Making GlusterFS Log Files Readable by Fluentd
By default, only root can read the GlusterFS log files. We'll allow others to read the file.
1
$ ls -alF /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
2
-rw------- 1 root root 1385  Feb  3 07:21 2014 /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
3
$ sudo chmod +r /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
4
$ ls -alF /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
5
-rw-r--r-- 1 root root 1385  Feb  3 07:21 2014 /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
Copied!
Now, modify Fluentd's configuration file. It is located at /etc/td-agent/td-agent.conf. `td-agent` is Fluentd's rpm/deb package maintained by Treasure Data​
This is what the configuration file should look like:
1
$ sudo cat /etc/td-agent/td-agent.conf
2
​
3
<source>
4
  @type glusterfs_log
5
  path /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
6
  pos_file /var/log/td-agent/etc-glusterfs-glusterd.vol.log.pos
7
  tag glusterfs_log.glusterd
8
  format /^(?<message>.*)$/
9
</source>
10
​
11
<match glusterfs_log.**>
12
  @type forward
13
  send_timeout 60s
14
  recover_wait 10s
15
  heartbeat_interval 1s
16
  phi_threshold 8
17
  hard_timeout 60s
18
​
19
  <server>
20
    name logserver
21
    host 172.31.10.100
22
    port 24224
23
    weight 60
24
  </server>
25
​
26
  <secondary>
27
    @type file
28
    path /var/log/td-agent/forward-failed
29
  </secondary>
30
</match>
Copied!
the ... section is for failover (when the aggregator instance at 172.31.10.100:24224 is unreachable).
Finally, start td-agent. Fluentd will started with the updated setup.
1
$ sudo service td-agent start
2
Starting td-agent:                                         [  OK  ]
Copied!
Step 3: Setting Up the Aggregator Fluentd Server
We'll now set up a separate Fluentd instance to aggregate the logs. Again, the first step is to install Fluentd.
1
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat.sh | sh
Copied!
We'll set up the node to send data to Elasticsearch, where the logs will be indexed and written to local disk for backup.
First, install the Elasticsearch output plugin as follows:
1
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-glusterfs
Copied!
Then, configure Fluentd as follows:
1
$ sudo cat /etc/td-agent/td-agent.conf
2
<source>
3
  @type forward
4
  port 24224
5
  bind 0.0.0.0
6
</source>
7
​
8
<match glusterfs_log.glusterd>
9
  @type copy
10
​
11
  #local backup
12
  <store>
13
    @type file
14
    path /var/log/td-agent/glusterd
15
  </store>
16
​
17
  #Elasticsearch
18
  <store>
19
    @type elasticsearch
20
    host ELASTICSEARCH_URL_HERE
21
    port 9200
22
    index_name glusterfs
23
    type_name fluentd
24
    logstash_format true
25
  </store>
26
</match>
Copied!
That's it! You should now be able to search and visualize your GlusterFS logs with Kibana.
Acknowledgement
This article is inspired by Daisuke Sasaki's article on Classmethod's website. Thanks Daisuke!
Learn More
​Fluentd Architecture​
​Fluentd Get Started​
​GlusterFS Input Plugin​
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.

Cep Norikra

Common Log Formats

Last modified 2yr ago

Copy link

Contents

Background

Setting up Fluentd on GlusterFS Nodes

Step 1: Installing Fluentd

Step 2: Making GlusterFS Log Files Readable by Fluentd

Step 3: Setting Up the Aggregator Fluentd Server

Acknowledgement

Learn More