This article shows how to use Fluentd to collect GlusterFS logs for analysis (search, analytics, troubleshooting, etc.)
GlusterFS is an open source, distributed file system commercially supported by Red Hat, Inc. Each node in GlusterFS generates its own logs, and it's sometimes convenient to have these logs collected in a central location for analysis (e.g., When one GlusterFS node went down, what was happening on other nodes?).
Fluentd is an open source data collector for high-volume data streams. It's a great fit for monitoring GlusterFS clusters because:
Fluentd supports GlusterFS logs as a data source.
Fluentd supports various output systems (e.g., Elasticsearch,
MongoDB, Treasure Data, etc.) that can help GlusterFS users analyze
The rest of this article explains how to set up Fluentd with GlusterFS. For this example, we chose Elasticsearch as the backend system.
First, we'll install Fluentd using the following command:
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
Next, we'll install the Fluentd plugin for GlusterFS:
$ sudo /usr/sbin/td-agent-gem install fluent-plugin-glusterfsFetching: fluent-plugin-glusterfs-1.0.0.gem (100%)Successfully installed fluent-plugin-glusterfs-1.0.01 gem installedInstalling ri documentation for fluent-plugin-glusterfs-1.0.0...Installing RDoc documentation for fluent-plugin-glusterfs-1.0.0...
By default, only
root can read the GlusterFS log files. We'll allow others to read the file.
$ ls -alF /var/log/glusterfs/etc-glusterfs-glusterd.vol.log-rw------- 1 root root 1385 Feb 3 07:21 2014 /var/log/glusterfs/etc-glusterfs-glusterd.vol.log$ sudo chmod +r /var/log/glusterfs/etc-glusterfs-glusterd.vol.log$ ls -alF /var/log/glusterfs/etc-glusterfs-glusterd.vol.log-rw-r--r-- 1 root root 1385 Feb 3 07:21 2014 /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
Now, modify Fluentd's configuration file. It is located at
/etc/td-agent/td-agent.conf. `td-agent` is Fluentd's rpm/deb package maintained by Treasure Data
This is what the configuration file should look like:
$ sudo cat /etc/td-agent/td-agent.conf<source>@type glusterfs_logpath /var/log/glusterfs/etc-glusterfs-glusterd.vol.logpos_file /var/log/td-agent/etc-glusterfs-glusterd.vol.log.postag glusterfs_log.glusterdformat /^(?<message>.*)$/</source><match glusterfs_log.**>@type forwardsend_timeout 60srecover_wait 10sheartbeat_interval 1sphi_threshold 8hard_timeout 60s<server>name logserverhost 172.31.10.100port 24224weight 60</server><secondary>@type filepath /var/log/td-agent/forward-failed</secondary></match>
the ... section is for failover (when the aggregator instance at 172.31.10.100:24224 is unreachable).
Finally, start td-agent. Fluentd will started with the updated setup.
$ sudo service td-agent startStarting td-agent: [ OK ]
We'll now set up a separate Fluentd instance to aggregate the logs. Again, the first step is to install Fluentd.
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat.sh | sh
We'll set up the node to send data to Elasticsearch, where the logs will be indexed and written to local disk for backup.
First, install the Elasticsearch output plugin as follows:
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-glusterfs
Then, configure Fluentd as follows:
$ sudo cat /etc/td-agent/td-agent.conf<source>@type forwardport 24224bind 0.0.0.0</source><match glusterfs_log.glusterd>@type copy#local backup<store>@type filepath /var/log/td-agent/glusterd</store>#Elasticsearch<store>@type elasticsearchhost ELASTICSEARCH_URL_HEREport 9200index_name glusterfstype_name fluentdlogstash_format true</store></match>
That's it! You should now be able to search and visualize your GlusterFS logs with Kibana.
This article is inspired by Daisuke Sasaki's article on Classmethod's website. Thanks Daisuke!
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.