Last updated
Last updated
This article explains how to collect logs to EFK (Elasticsearch + Fluentd + Kibana) stack. The example uses for setting up multiple containers.
is an open source search engine known for its ease of use. is an open source Web UI that makes Elasticsearch user friendly for marketers, engineers and data scientists alike.
By combining these three tools EFK (Elasticsearch + Fluentd + Kibana) we get a scalable, flexible, easy to use log collection and analytics pipeline. In this article, we will set up 4 containers, each includes:
All of httpd
's logs will be ingested into Elasticsearch + Kibana, via Fluentd.
Please download and install Docker / Docker Compose. Well, that's it :)
With the YAML file below, you can create and start all the services (in this case, Apache, Fluentd, Elasticsearch, Kibana) by one command.
Let's start all of the containers, with just one command.
You can check to see if 4 containers are running by docker ps
command.
Let's access to httpd
to generate some access logs. curl
command is always your friend.
Please go to http://localhost:5601/
with your browser. Then, you need to set up the index name pattern for Kibana. Please specify fluentd-*
to Index name or pattern
and press Create
button.
Then, go to Discover
tab to seek for the logs. As you can see, logs are properly collected into Elasticsearch + Kibana, via Fluentd.
This article explains how to collect logs from Apache to EFK (Elasticsearch + Fluentd + Kibana). The example code is available in this repository.
First, please prepare docker-compose.yml
for . Docker Compose is a tool for defining and running multi-container Docker applications.
logging
section (check ) of web
container specifies as a default container logging driver. All of the logs from web
container will be automatically forwarded to host:port specified by fluentd-address
.
Then, please prepare fluentd/Dockerfile
with the following content, to use Fluentd's and additionally install Elasticsearch plugin.
Then, please prepare Fluentd's configuration file fluentd/conf/fluent.conf
. plugin is used for receive logs from Docker logging driver, and out_elasticsearch is for forwarding logs to Elasticsearch.
If this article is incorrect or outdated, or omits critical information, please . is a open source project under . All components are available under the Apache 2 License.