Free Alternative To Splunk By Fluentd

​Splunk is a great tool for searching logs, but its high cost makes it prohibitive for many teams. In this article, we present a free and open source alternative to Splunk by combining three open source projects: Elasticsearch, Kibana, and Fluentd.
​Elasticsearch is an open source search engine known for its ease of use. Kibana is an open source Web UI that makes Elasticsearch user friendly for marketers, engineers and data scientists alike.
By combining these three tools (Fluentd + Elasticsearch + Kibana) we get a scalable, flexible, easy to use log search engine with a great Web UI that provides an open-source Splunk alternative, all for free.
In this guide, we will go over installation, setup, and basic use of this combined log search solution. This article was tested on Mac OS X Mountain Lion. If you're not familiar with Fluentd, please learn more about Fluentd first.
​What is Fluentd?​
Prerequisites
Java for Elasticsearch
Please confirm that your Java version is 8 or higher.
1
$ java -version
2
java version "1.8.0_111"
3
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
4
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
Copied!
Now that we've checked for prerequisites, we're now ready to install and set up the three open source tools.
Set Up Elasticsearch
To install Elasticsearch, please download and extract the Elasticsearch package as shown below.
1
$ curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.0.2.tar.gz
2
$ tar zxvf elasticsearch-5.0.2.tar.gz
3
$ cd elasticsearch-5.0.2
Copied!
Once installation is complete, start Elasticsearch.
1
$ ./bin/elasticsearch
Copied!
Set Up Kibana
To install Kibana, download it via the official webpage and extract it. Kibana is a HTML / CSS / JavaScript application. Download page is here. In this article, we download Mac OS X binary.
1
$ curl -O https://artifacts.elastic.co/downloads/kibana/kibana-5.0.2-darwin-x86_64.tar.gz
2
$ tar zxvf kibana-5.0.2-darwin-x86_64.tar.gz
3
$ cd kibana-5.0.2-darwin-x86_64
Copied!
Once installation is complete, start Kibana and run ./bin/kibana. You can modify Kibana's configuration via config/kibana.yml.
1
$ ./bin/kibana
Copied!
Access http://localhost:5601 in your browser.
Set Up Fluentd (td-agent)
In this guide We'll install td-agent, the stable release of Fluentd. Please refer to the guides below for detailed installation steps.
​Debian Package​
​RPM Package​
​Ruby gem​
Next, we'll install the Elasticsearch plugin for Fluentd: fluent-plugin-elasticsearch. Then, install fluent-plugin-elasticsearch as follows.
1
$ sudo /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch --no-document
Copied!
We'll configure td-agent (Fluentd) to interface properly with Elasticsearch. Please modify /etc/td-agent/td-agent.conf as shown below:
1
# get logs from syslog
2
<source>
3
  @type syslog
4
  port 42185
5
  tag syslog
6
</source>
7
​
8
# get logs from fluent-logger, fluent-cat or other fluentd instances
9
<source>
10
  @type forward
11
</source>
12
​
13
<match syslog.**>
14
  @type elasticsearch
15
  logstash_format true
16
  flush_interval 10s # for testing
17
</match>
Copied!
fluent-plugin-elasticsearch comes with a logstash_format option that allows Kibana to search stored event logs in Elasticsearch.
Once everything has been set up and configured, we'll start td-agent.
1
$ sudo /etc/init.d/td-agent start
Copied!
Set Up rsyslogd
In our final step, we'll forward the logs from your rsyslogd to Fluentd. Please add the following line to your /etc/rsyslog.conf, and restart rsyslog. This will forward your local syslog to Fluentd, and Fluentd in turn will forward the logs to Elasticsearch.
1
*.* @127.0.0.1:42185
Copied!
Please restart the rsyslog service once the modification is complete.
1
$ sudo /etc/init.d/rsyslog restart
Copied!
Store and Search Event Logs
Once Fluentd receives some event logs from rsyslog and has flushed them to Elasticsearch, you can search the stored logs using Kibana by accessing Kibana's index.html in your browser. Here is an image example.
To manually send logs to Elasticsearch, please use the logger command.
1
$ logger -t test foobar
Copied!
When debugging your td-agent configuration, using filter_stdout will be useful. All the logs including errors can be found at /etc/td-agent/td-agent.log.
1
<filter syslog.**>
2
  @type stdout
3
</filter>
4
​
5
<match syslog.**>
6
  @type elasticsearch
7
  logstash_format true
8
  flush_interval 10s # for testing
9
</match>
Copied!
Conclusion
This article introduced the combination of Fluentd and Kibana (with Elasticsearch) which achieves a free alternative to Splunk: storing and searching machine logs. The examples provided in this article have not been tuned.
If you will be using these components in production, you may want to modify some of the configurations (e.g. JVM, Elasticsearch, Fluentd buffer, etc.) according to your needs.
Learn More
​Fluentd Architecture​
​Fluentd Get Started​
​Downloading Fluentd​
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.

Forwarding Over Ssl

Data Collection to Hadoop (HDFS)

Last modified 2yr ago

Copy link

Contents

Prerequisites

Java for Elasticsearch

Set Up Elasticsearch

Set Up Kibana

Set Up Fluentd (td-agent)

Set Up rsyslogd

Store and Search Event Logs

Conclusion

Learn More