This article explains how to collect Docker logs and propagate them to EFK (Elasticsearch + Fluentd + Kibana) stack. The example uses Docker Compose for setting up multiple containers.
Elasticsearch is an open-source search engine known for its ease of use. Kibana is an open-source Web UI that makes Elasticsearch user-friendly for marketers, engineers and data scientists alike.
By combining these three tools EFK (Elasticsearch + Fluentd + Kibana) we get a scalable, flexible, easy to use log collection and analytics pipeline. In this article, we will set up four (4) containers, each includes:
All the logs of httpd
will be ingested into Elasticsearch + Kibana, via Fluentd.
Please download and install Docker / Docker Compose. Well, that's it :)
Create docker-compose.yml
for Docker Compose. Docker Compose is a tool for defining and running multi-container Docker applications.
With the YAML file below, you can create and start all the services (in this case, Apache, Fluentd, Elasticsearch, Kibana) by one command:
version: '3'services:web:image: httpdports:- "80:80"links:- fluentdlogging:driver: "fluentd"options:fluentd-address: localhost:24224tag: httpd.accessfluentd:build: ./fluentdvolumes:- ./fluentd/conf:/fluentd/etclinks:- "elasticsearch"ports:- "24224:24224"- "24224:24224/udp"elasticsearch:image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2environment:- "discovery.type=single-node"expose:- "9200"ports:- "9200:9200"kibana:image: kibana:7.10.1links:- "elasticsearch"ports:- "5601:5601"
The logging
section (check Docker Compose documentation) of web
container specifies Docker Fluentd Logging Driver as a default container logging driver. All the logs from the web
container will automatically be forwarded to host:port
specified by fluentd-address
.
Create fluentd/Dockerfile
with the following content using the Fluentd official Docker image; and then, install the Elasticsearch plugin:
# fluentd/DockerfileFROM fluent/fluentd:v1.12.0-debian-1.0USER rootRUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "4.3.3"]USER fluent
Then, create the Fluentd configuration file fluentd/conf/fluent.conf
. The forward
input plugin receives logs from the Docker logging driver and elasticsearch
output plugin forwards these logs to Elasticsearch.
# fluentd/conf/fluent.conf<source>@type forwardport 24224bind 0.0.0.0</source><match *.**>@type copy<store>@type elasticsearchhost elasticsearchport 9200logstash_format truelogstash_prefix fluentdlogstash_dateformat %Y%m%dinclude_tag_key truetype_name access_logtag_key @log_nameflush_interval 1s</store><store>@type stdout</store></match>
Let's start the containers:
$ docker-compose up
Use docker ps
command to verify that the four (4) containers are up and running:
$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES558fd18fa2d4 httpd "httpd-foreground" 17 seconds ago Up 16 seconds 0.0.0.0:80->80/tcp docker_web_1bc5bcaedb282 kibana:7.10.1 "/usr/local/bin/kiba…" 18 seconds ago Up 17 seconds 0.0.0.0:5601->5601/tcp docker_kibana_19fe2d02cff41 docker.elastic.co/elasticsearch/elasticsearch:7.10.2 "/usr/local/bin/dock…" 20 seconds ago Up 18 seconds 0.0.0.0:9200->9200/tcp, 9300/tcp docker_elasticsearch_1
Use curl
command to generate some access logs like this:
$ curl http://localhost:80/[1-10]<html><body><h1>It works!</h1></body></html><html><body><h1>It works!</h1></body></html><html><body><h1>It works!</h1></body></html><html><body><h1>It works!</h1></body></html><html><body><h1>It works!</h1></body></html><html><body><h1>It works!</h1></body></html><html><body><h1>It works!</h1></body></html><html><body><h1>It works!</h1></body></html><html><body><h1>It works!</h1></body></html><html><body><h1>It works!</h1></body></html>
Browse to http://localhost:5601/app/management/kibana/indexPatterns
and set up the index name pattern for Kibana. Specify fluentd-*
to Index name or pattern
and click Create
.
Then, go to Discover
tab to check the logs. As you can see, logs are properly collected into the Elasticsearch + Kibana, via Fluentd.
The code is available at https://github.com/digikin/fluentd-elastic-kibana.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.