Versions | v0.12 (td-agent2) | v0.10 (td-agent1)

This page is for v0.10, not the latest stable version which is v0.12. For the latest stable version of this article, click here.


Common Log Formats and How To Parse Them

This page is a glossary of common log formats that can be parsed with the Tail input plugin.

  • Apache Access Log

    Use format apache2 as shown below:

      <source>
          type tail
          format apache2
          tag apache.access
          path /var/log/apache2/access.log
      </source>
    
  • Apache Error Log

    Use a regular expression. See the format field in the following sample configuration.

      <source>
          type tail
          format /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\] \[pid (?<pid>[^\]]*)\] \[client (?<client>[^\]]*)\] (?<message>.*)$/
          tag apache.error
          path /var/log/apache2/error.log
      </source>
    

    Depending on your particular error log format, you may need to adjust the regular expression above. You can test your format using fluentd-ui’s in_tail editor or Fluentular.

  • Maillog

    Use a regular expression. See the format field in the following sample configuration.

      <source>
          type tail
          format /^(?<time>[^ ]+) (?<host>[^ ]+) (?<process>[^:]+): (?<message>((?<key>[^ :]+)[ :])? ?((to|from)=<(?<address>[^>]+)>)?.*)$/
          tag postfix.maillog
          path /var/log/maillog
      </source>
    
  • Nginx Access Log

    Use format nginx as shown below:

      <source>
          type tail
          format nginx
          tag nginx.access
          path /var/log/nginx/access.log
      </source>
    
  • Nginx Error Log

    Use the format* and multiline_flush_interval fields in the following sample configuration. Applications running under Nginx can output multi-line errors including stack traces, so the multiline mode is a good fit.

      <source>
          type tail
          tag nginx.error
          path /var/log/nginx/error.log
    
          format multiline
          format_firstline /^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \[\w+\] (?<pid>\d+).(?<tid>\d+): /
          format1 /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)/
          multiline_flush_interval 3s
      </source>
    

    If you know your error log will only contain single lines, you can use the below simpler configuration with just a format.

      <source>
          type tail
          format /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)$/
          tag nginx.error
          path /var/log/nginx/error.log
      </source>
    
  • GlusterFS Logs

    Use the GlusterFS input plugin.

Do you not see what you are looking for?

Give us a shout on GitHub, Twitter or the mailing list. Better yet, we always welcome a pull request!

Last updated: 2016-06-06 04:47:13 UTC

Versions | v0.12 (td-agent2) | v0.10 (td-agent1)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF), originally invented by Treasure Data, Inc. All components are available under the Apache 2 License.

Interested in the Fluentd Newsletters?