Fluentd
Search…
Send Syslog Data to Graylog
This article explains how to set up Fluentd with Graylog. Graylog is a popular log management server powered by Elasticsearch and MongoDB. You can combine Fluentd and Graylog to create a scalable log analytics pipeline.
Prerequisites
- Basic Understanding of Fluentd
- Linux Server (Ubuntu 18.04 LTS was used for this guide.)
How to Setup Graylog + Fluentd
Dependencies
Install the dependencies with the following command:
1
$ sudo apt install mongodb-server openjdk-8-jre-headless uuid-runtime
Copied!
Elasticsearch
Graylog requires Elasticsearch, which can be instantly launched using the following commands:
1
# Note that Graylog 2.4 doesn't support Elasticsearch 6.X.
2
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.13.tar.gz
3
$ tar -xzf elasticsearch-5.6.13.tar.gz
4
$ cd elasticsearch-5.6.13
Copied!
Elasticsearch is ready. Start it with:
1
$ ./bin/elasticsearch
Copied!
Graylog
In this article, we will use Graylog 2.4.
1
$ wget https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.deb
2
$ sudo dpkg -i graylog-2.4-repository_latest.deb
Copied!
Update the package cache and install
graylog-server:1
$ sudo apt update
2
$ sudo apt install graylog-server
Copied!
Open
/etc/graylog/server/server.conf and configure the following parameters:- 1.
password_secret - 2.
root_password_sha2. - 3.
web_enable
For
root_password_sha2, run echo -n ROOT_PASSWORD | sha256sum and set the hash. Also you need to set web_enable to true to access the web interface.Now let's start Graylog:
1
$ sudo systemctl start graylog-server
Copied!
Prepare Graylog for Fluentd
To log in, use
admin as the username and YOUR_PASSWORD as the password (the one you have set up for root_password_sha2).Once logged in, click on
System in the top nav. Next, click on Inputs from the left navigation bar. (Or, simply go to http://localhost:9000/system/inputs.Then, from the dropdown, choose
GELF UDP and click on Launch new input, which should pop up a modal dialogue, Select the Node and fill the Title. Then, click Save.
Graylog Inputs
Now, Graylog2 is ready to accept messages from Fluentd over UDP. It is time to configure Fluentd.
Fluentd
1
$ wget http://packages.treasuredata.com.s3.amazonaws.com/3/ubuntu/bionic/pool/contrib/t/td-agent/td-agent_3.2.1-0_amd64.deb
2
$ sudo dpkg -i td-agent_3.2.1-0_amd64.deb
Copied!
Then, install the
out_gelf plugin to send data to Graylog. Currently, the GELF plugin is not available on RubyGems, so we need to download the plugin file and place it in /etc/td-agent/plugin:1
$ wget https://raw.githubusercontent.com/emsearcy/fluent-plugin-gelf/master/lib/fluent/plugin/out_gelf.rb
2
$ sudo mv out_gelf.rb /etc/td-agent/plugin
Copied!
We also need to gem-install GELF's Ruby client:
1
$ sudo /usr/sbin/td-agent-gem install gelf
Copied!
Configure
/etc/td-agent/td-agent.conf as follows:1
<source>
2
@type syslog
3
tag graylog2
4
</source>
5
​
6
<match graylog2.**>
7
@type gelf
8
host 127.0.0.1
9
port 12201
10
<buffer>
11
flush_interval 5s
12
</buffer>
13
</match>
Copied!
Open
/etc/rsyslog.conf and add the following line to the file:1
*.* @127.0.0.1:5140
Copied!
Finally, restart
rsyslog and Fluentd with the following commands:1
$ sudo systemctl restart rsyslog
2
$ sudo systemctl restart td-agent
Copied!
Visualize the Data Stream
When you log back into Graylog, you should be seeing a graph like this (wait for events to flow in):
Graylog Graph
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 11mo ago