Docker Logging Driver
Last updated
Was this helpful?
Last updated
Was this helpful?
The article describes how to implement a unified logging system for your containers. An application in a production environment requires to register certain events or problems during its runtime.
The old-fashioned way is to write these messages into a log file, but that inherits certain problems. Specifically, when we try to perform some analysis over the registers, or on the other hand, if the application has multiple instances running, the scenario becomes even more complex.
On Docker v1.6, the concept of was introduced. The Docker engine is aware of the output interfaces that manage the application messages.
For Docker v1.8, we have implemented a native . Now, you are able to have a unified and structured logging system with the simplicity and high performance of .
NOTE: Currently, the Fluentd logging driver doesn't support sub-second precision.
Using the Docker logging mechanism with is a straightforward step. To get started, make sure you have the following prerequisites:
A basic understanding of
A basic understanding of Docker
A basic understanding of
Docker v1.8+
For simplicity, the Fluentd is launched as a standard process, not as a container.
Please refer to for a fully containerized tutorial.
The first step is to prepare Fluentd to listen for the messages coming from the Docker containers. For demonstration purposes, we will instruct Fluentd to write the messages to the standard output. Later, you will find how to accomplish the same by aggregating the logs into a MongoDB instance.
Create demo.conf with the following configuration:
Now, start an instance of Fluentd like this:
On successful start, you should see the Fluentd startup logs:
By default, the Fluentd logging driver will try to find a local Fluentd instance (Step # 2) listening for connections on the TCP port 24224. Note that the container will not start if it cannot connect to the Fluentd instance.
The following command will run a base Ubuntu container and print some messages to the standard output:
Note that we have launched the container specifying the Fluentd logging driver i.e. --log-driver=fluentd.
Now, you should see the incoming messages from the container in Fluentd logs:
At this point, you will notice that the incoming messages are in JSON format, have a timestamp, are tagged with the container_id and contain general information from the source container along with the message.
Then you provide the log message with JSON format:
Original Event (without filter plugin):
Filtered Event:
At first, you need to create custom docker image due to install the fluent-plugin-concat gem in the Fluentd container.
Create Dockerfile with the following content:
Build the custom image:
Then, create the configuration file demo.conf with the following content:
Launch the Fluentd container:
Then you provide the log message contains newlines:
Original Events (without filter plugin):
Filtered Events:
NOTE: For plugins with the simple file structure, such as fluent-plugin-concat, plugins directory can be used instead of creating custom docker image.
Prepare the plugins directory and copy the plugin file:
Launch the Fluentd container with plugins directory mounted:
fluentd-address
tag
fluentd-addressSpecifies the optional address (<ip>:<port>) for Fluentd.
Example:
tagAdditionally, this option allows to specify some internal variables such as {{.ID}}, {{.FullID}} or {{.Name}} like this:
For a real-world use-case, you would want to use something other than the Fluentd standard output to store Docker container messages, such as Elasticsearch, MongoDB, HDFS, S3, Google Cloud Storage, and so on.
This document describes how to set up a multi-container logging environment via EFK (Elasticsearch, Fluentd, Kibana) with Docker Compose.
In a production environment, you must use one of the container orchestration tools. Currently, Kubernetes has better integration with Fluentd, and we're working on making better integrations with other tools as well.
The application log is stored in the "log" field in the record. You can parse this log before sending it to the destinations by using .
About --log-opt tag=..., please refer at section.
The application log is stored in the log field of the record. You can concatenate these logs by using filter before sending it to the destinations.
If the logs are typical stacktraces, consider using plugin instead.
The supports following options through the --log-opt Docker command-line argument:
are a major requirement for Fluentd as they allow for identifying the source of incoming data and take routing decisions. By default, the Fluentd logging driver uses the container_id as a tag (64 character ID). You can change its value with the tag option like this:
If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.