# Logging This article describes Fluentd's logging mechanism. Fluentd has two log layers: global and per plugin. Different log levels can be set for global logging and plugin level logging. ## Log Level Shown below is the list of supported values, in increasing order of verbosity: * `fatal` * `error` * `warn` * `info` * `debug` * `trace` The default log level is `info`, and Fluentd outputs `info`, `warn`, `error` and `fatal` logs by default. ## Global Logs Global logging is used by Fluentd core and plugins that don't set their own log levels. The global log level can be adjusted up or down. ### By Command Line Option #### Increase Verbosity Level The `-v` option sets the verbosity to `debug` while the `-vv` option sets the verbosity to `trace`. ``` $ fluentd -v ... # debug level $ fluentd -vv ... # trace level ``` These options are useful for debugging purposes. #### Decrease Verbosity Level The `-q` option sets the verbosity to `warn` while the `-qq` option sets the verbosity to `error`. ``` $ fluentd -q ... # warn level $ fluentd -qq ... # error level ``` ### By Config File You can also change the logging level with `` section in the config file like below. ``` # equal to -qq option log_level error ``` ## Per Plugin Log The `log_level` option sets different levels of logging for each plugin. It can be set in each plugin's configuration file. For example, in order to debug [in\_tail](broken://pages/-LR7PNxnK7pp6OKiK7fu) but suppress all but fatal log messages for [in\_http](broken://pages/-LR7PNxshWu5hf4ObEp-), their respective `log_level` options should be set as follows: ``` @type tail @log_level debug path /var/log/data.log ... @type http @log_level fatal ``` If you don't specify the `log_level` parameter, the plugin will use the global log level. Some plugins haven't supported per-plugin logging yet. The [logging section of the Plugin Development article](/0.12/developer/plugin-development.md#logging) explains how to update such plugins to support the new log level system. ## Suppress repeated stacktrace Fluentd can suppress same stacktrace with `--suppress-repeated-stacktrace`. For example, if you pass `--suppress-repeated-stacktrace` to fluentd: ``` 2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:154:rescue in emit_stream: emit transaction failed error_class = RuntimeError error = # 2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/out_stdout.rb:43:in `emit' [snip] 2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/in_object_space.rb:63:in `run' 2013-12-04 15:05:53 +0900 [error]: plugin/in_object_space.rb:113:rescue in on_timer: object space failed to emit error = "foo.bar" error_class = "RuntimeError" tag = "foo" record = "{ ...}" 2013-12-04 15:05:55 +0900 [warn]: fluent/engine.rb:154:rescue in emit_stream: emit transaction failed error_class = RuntimeError error = # 2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/o/2.0.0/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run' [snip] ``` logs are changed to: ``` 2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:154:rescue in emit_stream: emit transaction failed error_class = RuntimeError error = # 2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/o/2.0.0/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run' [snip] 2013-12-04 15:05:53 +0900 [warn]: fluent/engine.rb:140:emit_stream: /Users/repeatedly/devel/fluent/fluentd/lib/fluent/plugin/in_object_space.rb:63:in `run' 2013-12-04 15:05:53 +0900 [error]: plugin/in_object_space.rb:113:rescue in on_timer: object space failed to emit error = "foo.bar" error_class = "RuntimeError" tag = "foo" record = "{ ...}" 2013-12-04 15:05:55 +0900 [warn]: fluent/engine.rb:154:rescue in emit_stream: emit transaction failed error_class = RuntimeError error = # 2013-12-04 15:05:55 +0900 [warn]: plugin/in_object_space.rb:111:on_timer: suppressed same stacktrace ``` Same stacktrace is replaced with `suppressed same stacktrace` message until other stacktrace is received. ## Output to log file Fluentd outputs logs to `STDOUT` by default. To output to a file instead, please specify the `-o` option. ``` $ fluentd -o /path/to/log_file ``` Fluentd doesn't support log rotation yet. ## Capture Fluentd logs Fluentd marks its own logs with the `fluent` tag. You can process Fluentd logs by using `` or ``(Of course, `**` captures other logs). If you define `` in your configuration, then Fluentd will send its own logs to this match destination. This is useful for monitoring Fluentd logs. For example, if you have the following ``: ``` # omit other source / match @type stdout ``` then Fluentd outputs `fluent.info` logs to stdout like below: ``` 2014-02-27 00:00:00 +0900 [info]: shutting down fluentd 2014-02-27 00:00:01 +0900 fluent.info: {"message":"shutting down fluentd"} # by 2014-02-27 00:00:01 +0900 [info]: process finished code = 0 ``` ### Case1: Send Fluentd logs to monitoring service You can send Fluentd logs to a monitoring service by plugins, e.g. datadog, sentry, irc, etc. ``` # Add hostname for identifying the server @type record_transformer host "#{Socket.gethostname}" @type monitoring_plugin # parameters... ``` ### Case2: Use aggregation/monitoring server You can use [out\_forward](broken://pages/-LS2EqvLLGnjfrxO0sXh) to send Fluentd logs to a monitoring server. The monitoring server can then filter and send the logs to your notification system: chat, irc, etc. Leaf server example: ``` # Add hostname for identifying the server and tag to filter by log level @type record_transformer host "#{Socket.gethostname}" original_tag ${tag} @type forward # Monitoring server parameters ``` Monitoring server example: ``` @type forward label @FLUENTD_INTERNAL_LOG ``` If an error occurs, you will get a notification message in your irc `notify` channel. ``` 01:01 fluentd: [11:10:24] notice: fluent.warn [2014/02/27 01:00:00] @leaf.server.domain detached forwarding server 'server.name' ``` If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is a open source project under [Cloud Native Computing Foundation (CNCF)](https://cncf.io/). All components are available under the Apache 2 License. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.fluentd.org/0.12/deployment/logging.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.