# Store Apache Logs into MongoDB

This article explains how to use [Fluentd](http://fluentd.org/)'s MongoDB Output plugin ([out\_mongo](broken://pages/-LS2EqvW0RVW2fzsSUZq)) to aggregate semi-structured logs in real-time.

## Background

[Fluentd](http://fluentd.org/) is an advanced open-source log collector originally developed at [Treasure Data, Inc](http://www.treasuredata.com/). Because Fluentd handles logs as semi-structured data streams, the ideal database should have strong support for semi-structured data. There are several candidates that meet this criterion, but we believe [MongoDB](http://www.mongodb.org/) is the market leader.

MongoDB is an open-source, document-oriented database developed at [10gen, Inc](http://www.10gen.com/). It is schema-free and uses a JSON-like format to manage semi-structured data.

This article will show you how to use [Fluentd](http://fluentd.org/) to import Apache logs into MongoDB.

## Mechanism

The figure below shows how things will work.

![](/files/-LenuEFix8KK9dBfU0St)

Fluentd does 3 things:

1. It continuously "tails" the access log file.
2. It parses the incoming log entries into meaningful fields (such as

   `ip`, `path`, etc.) and buffers them.
3. It writes the buffered data to MongoDB periodically.

## Install

For simplicity, this article will describe how to set up an one-node configuration. Please install the following software on the same node.

* [Fluentd](http://fluentd.org/)
* [MongoDB Output Plugin](broken://pages/-LS2EqvW0RVW2fzsSUZq)
* [MongoDB](http://www.mongodb.org/)
* Apache (with the Combined Log Format)

The MongoDB Output plugin is included in the latest version of Fluentd's deb/rpm package. If you want to use Ruby Gems to install the plugin, please use `gem install fluent-plugin-mongo`.

* [Debian Package](/0.12/articles/install-by-deb.md)
* [RPM Package](/0.12/articles/install-by-rpm.md)
* [Ruby gem](/0.12/articles/install-by-gem.md)

For MongoDB, please refer to the following downloads page.

* [MongoDB Downloads](http://www.mongodb.org/downloads)

## Configuration

Let's start configuring Fluentd. If you used the deb/rpm package, Fluentd's config file is located at /etc/td-agent/td-agent.conf. Otherwise, it is located at /etc/fluentd/fluentd.conf.

### Tail Input

For the input source, we will set up Fluentd to track the recent Apache logs (typically found at /var/log/apache2/access\_log) The Fluentd configuration file should look like this:

```
<source>
  @type tail
  format apache2
  path /var/log/apache2/access_log
  pos_file /var/log/td-agent/apache2.access_log.pos
  tag mongo.apache.access
</source>
```

Please make sure that your Apache outputs are in the default \\'combined\\' format. \`format apache2\` cannot parse custom log formats. Please see the [in\_tail](broken://pages/-LR7PNxnK7pp6OKiK7fu) article for more information.

Let's go through the configuration line by line.

1. `type tail`: The tail Input plugin continuously tracks the log file.

   This handy plugin is included in Fluentd's core.
2. `format apache2`: Uses Fluentd's built-in Apache log parser.
3. `path /var/log/apache2/access_log`: The location of the Apache log.

   This may be different for your particular system.
4. `tag mongo.apache.access`: `mongo.apache.access` is used as the tag

   to route the messages within Fluentd.

That's it! You should now be able to output a JSON-formatted data stream for Fluentd to process.

### MongoDB Output

The output destination will be MongoDB. The output configuration should look like this:

```
<match mongo.**>
  # plugin type
  @type mongo

  # mongodb db + collection
  database apache
  collection access

  # mongodb host + port
  host localhost
  port 27017

  # interval
  flush_interval 10s

  # make sure to include the time key
  include_time_key true
</match>
```

The match section specifies the regexp used to look for matching tags. If a matching tag is found in a log, then the config inside `<match>...</match>` is used (i.e. the log is routed according to the config inside). In this example, the `mongo.apache.access` tag (generated by `tail`) is always used.

The `**` in `mongo.**` matches zero or more period-delimited tag parts (e.g. mongo/mongo.a/mongo.a.b).

**flush\_interval** specifies how often the data is written to MongoDB. The other options specify MongoDB's host, port, db, and collection.

For additional configuration parameters, please see the [MongoDB Output plugin](broken://pages/-LS2EqvW0RVW2fzsSUZq) article. If you are using ReplicaSet, please see the [MongoDB ReplicaSet Output plugin](broken://pages/-LS2EqvXidlzvnp184Kn) article.

## Test

To test the configuration, just ping the Apache server. This example uses the `ab` (Apache Bench) program.

```
$ ab -n 100 -c 10 http://localhost/
```

Then, access MongoDB and see the stored data.

```
$ mongo
> use apache
> db["access"].findOne();
{ "_id" : ObjectId("4ed1ed3a340765ce73000001"), "host" : "127.0.0.1", "user" : "-", "method" : "GET", "path" : "/", "code" : "200", "size" : "44", "time" : ISODate("2011-11-27T07:56:27Z") }
{ "_id" : ObjectId("4ed1ed3a340765ce73000002"), "host" : "127.0.0.1", "user" : "-", "method" : "GET", "path" : "/", "code" : "200", "size" : "44", "time" : ISODate("2011-11-27T07:56:34Z") }
{ "_id" : ObjectId("4ed1ed3a340765ce73000003"), "host" : "127.0.0.1", "user" : "-", "method" : "GET", "path" : "/", "code" : "200", "size" : "44", "time" : ISODate("2011-11-27T07:56:34Z") }
```

## Conclusion

Fluentd + MongoDB makes real-time log collection simple, easy, and robust.

## Learn More

* [Fluentd Architecture](https://www.fluentd.org/architecture)
* [Fluentd Get Started](broken://pages/-LkMNu6od_31-nAGsRGD)
* [MongoDB Output Plugin](broken://pages/-LS2EqvW0RVW2fzsSUZq)
* [MongoDB ReplicaSet Output Plugin](broken://pages/-LS2EqvXidlzvnp184Kn)

If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is a open source project under [Cloud Native Computing Foundation (CNCF)](https://cncf.io/). All components are available under the Apache 2 License.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.fluentd.org/0.12/articles/apache-to-mongodb.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.