Fluentd
Search…
Docker Logging
The following article describes how to implement an unified logging system for your Docker containers. Any production application requires to register certain events or problems during runtime.
The old fashion way is to write these messages to a log file, but that inherits certain problems specifically when we try to perform some analysis over the registers, or in the other side, if the application have multiple instances running, the escenario becomes even more complex.
On Docker v1.6, the concept of logging drivers was introduced, basically the Docker engine is aware about output interfaces that manage the application messages.
Could not load image
For Docker v1.8, we have implemented a native Fluentd Docker logging driver, now you are able to have an unified and structured logging system with the simplicity and high performance Fluentd. Currently, fluentd logging driver doesn't support sub-second precision.
Getting Started
Using the Docker logging mechanism with Fluentd is a straighforward step, to get started make sure you have the following prerequisites:
- A basic undersatnding of Docker
- Docker v1.8+
This article launches Fluentd as standard process, not a container. Please refer Docker Logging via EFK (Elasticsearch + Fluentd + Kibana) Stack with Docker Compose for fully containerized environment tutorial.
Step 1: Create the Fluentd configuration file
The first step is to prepare Fluentd to listen for the messsages that will receive from the Docker containers, for a demonstration purposes we will instruct Fluentd to write the messages to the standard output; In a later step you will find how to accomplish the same aggregating the logs into a MongoDB instance.
Create a simple file called in_docker.conf which contains the following entries:
1
<source>
2
type forward
3
port 24224
4
bind 0.0.0.0
5
</source>
6
​
7
<match *.*>
8
type stdout
9
</match>
Copied!
Step 2: Start Fluentd
With this simple command start an instance of Fluentd:
1
$ fluentd -c in_docker.conf
Copied!
If the service started you should see an output like this:
1
$ fluentd -c in_docker.conf
2
2015-09-01 15:07:12 -0600 [info]: reading config file path="in_docker.conf"
3
2015-09-01 15:07:12 -0600 [info]: starting fluentd-0.12.15
4
2015-09-01 15:07:12 -0600 [info]: gem 'fluent-plugin-mongo' version '0.7.10'
5
2015-09-01 15:07:12 -0600 [info]: gem 'fluentd' version '0.12.15'
6
2015-09-01 15:07:12 -0600 [info]: adding match pattern="*.*" type="stdout"
7
2015-09-01 15:07:12 -0600 [info]: adding source type="forward"
8
2015-09-01 15:07:12 -0600 [info]: using configuration file: <ROOT>
9
<source>
10
@type forward
11
port 24224
12
bind 0.0.0.0
13
</source>
14
<match docker.*>
15
@type stdout
16
</match>
17
</ROOT>
18
2015-09-01 15:07:12 -0600 [info]: listening fluent socket on 0.0.0.0:24224
Copied!
Step 3: Start Docker container with Fluentd driver
By default, the Fluentd logging driver will try to find a local Fluentd instance (step #2) listening for connections on the TCP port 24224, note that the container will not start if it cannot connect to the Fluentd instance.
Could not load image
The following command will run a base Ubuntu container and print some messages to the standard output, note that we have launched the container specifying the Fluentd logging driver:
1
$ docker run --log-driver=fluentd ubuntu echo "Hello Fluentd!"
2
Hello Fluentd!
Copied!
Step 4: Confirm
Now on the Fluentd output, you will see the incoming message from the container, e.g:
1
2015-09-01 15:10:40 -0600 docker.3fd8678d487e: {"source":"stdout","log":"Hello Fluentd!","container_id":"3fd8678d487e540c7a303e1613101e746c5012f3317434eda93f24351c1928f7","container_name":"/angry_kalam"}
Copied!
At this point you will notice something interesting, the incoming messages have a timestamp, are tagged with the container_id and contains general information from the source container along the message, everything in JSON format.
Additional Step 1: Parse log message
Application log is stored into
"log" field in the record. You can parse this log by using filter_parser filter before send to destinations.1
<filter docker.**>
2
@type parser
3
format json # apache2, nginx, etc...
4
key_name log
5
reserve_data true
6
</filter>
Copied!
Original event:
1
2015-09-01 15:10:40 -0600 docker.3fd8678d487e: {"source":"stdout","log":"{\"key\":\"value\"}","container_id":"3fd8678d487e540c7a303e1613101e746c5012f3317434eda93f24351c1928f7","container_name":"/angry_kalam"}
Copied!
Filtered event:
1
2015-09-01 15:10:40 -0600 docker.3fd8678d487e: {"source":"stdout","log":"{\"key\":\"value\"}","container_id":"3fd8678d487e540c7a303e1613101e746c5012f3317434eda93f24351c1928f7","container_name":"/angry_kalam","key":"value"}
Copied!
Additional Step 2: Concatenate multiple lines log messages
Application log is stored into
"log" field in the records. You can concatenate these logs by using fluent-plugin-concat filter before send to destinations.1
<filter docker.**>
2
@type concat
3
key log
4
stream_identity_key container_id
5
multiline_start_regexp /^-e:2:in `\/'/
6
multiline_end_regexp /^-e:4:in/
7
</filter>
Copied!
Original events:
1
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'"}
2
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:2:in `do_division_by_zero'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"}
3
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:4:in `<main>'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"}
Copied!
Filtered events:
1
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'\n-e:2:in `do_division_by_zero'\n-e:4:in `<main>'"}
Copied!
Driver options
- fluentd-address
- tag
fluentd-address
Specify an optional address for Fluentd, it allows to set the host and TCP port, e.g:
1
$ docker run --log-driver=fluentd --log-opt fluentd-address=192.168.2.4:24225 ubuntu echo "..."
Copied!
tag
​Tags are a major requirement on Fluentd, they allows to identify the incoming data and take routing decisions. By default the Fluentd logging driver uses the container_id as a tag (64 character ID), you can change it value with the tag option as follows:
1
$ docker run --log-driver=fluentd --log-opt tag=docker.my_new_tag ubuntu echo "..."
Copied!
Additionally this option allows to specify some internal variables: {{.ID}}, {{.FullID}} or {{.Name}}. e.g:
1
$ docker run --log-driver=fluentd --log-opt tag=docker.{{.ID}} ubuntu echo "..."
Copied!
Development Environments
In a more real-world use case, you would want to use something other than the Fluentd standard output to store Docker containers messages, such as Elasticsearch, MongoDB, HDFS, S3, Google Cloud Storage and so on.
This document describes how to set up multi-container logging environment via EFK (Elasticsearch, Fluentd, Kibana) with Docker Compose.
Production Environments
In production environment, you must use one of the container orchestration tools. Currently, Kubernetes has better integration with Fluentd, and we're working on making better integrations with other tools as well.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 2yr ago
Copy link
Contents
Getting Started
Step 1: Create the Fluentd configuration file
Step 2: Start Fluentd
Step 3: Start Docker container with Fluentd driver
Step 4: Confirm
Additional Step 1: Parse log message
Additional Step 2: Concatenate multiple lines log messages
Driver options
Development Environments
Production Environments