Fluentd is an open source software to collect events and logs in JSON format. It has hundred of plugins that allows you to store the logs/events in your favorite data store like AWS S3, MongoDB and even elasticsearch. A CloudStack plugin has been written to be able to listen to CloudStack events and store these events in a chosen storage backend. In this chapter we will show how to store CloudStack logs in MongoDB using Fluentd. The documentation quite straightforward, but here are the basic steps.
You will need a working fluentd installed on your machine. Pick your package manager of choice and install fluentd, for instance with gem we would do:
sudo gem install fluentd
fluentd will now be in your path, you need to create a configuration file and start fluentd using this config. For additional options with fluentd just enter fluentd -h. The s option will create a sample configuration file in the working directory. The -c option will start fluentd using the specific configuration file. You can then send a test log/event message to the running process with fluent-cat
CloudStack has a listEvents API which does what is says :) it lists events happening within a CloudStack deployment. Such events as the start and stop of a virtual machine, creation of security groups, life cycles events of storage elements, snapshots etc. The listEvents API is well documented. Based mostly on this API and the fog ruby library, a CloudStack plugin for fluentd was written by Yuichi UEMURA. It is slightly different from using logstash, as with logstash you can format the log4j logs of the CloudStack management server and directly collect those. Here we rely on the listEvents API.
Then build your own gem and install it with sudo gem build fluent-plugin-cloudstack.gemspec and sudo gem install fluent-plugin-cloudstack-0.0.8.gem
Or you install the gem directly:
sudo gem install fluent-plugin-cloudstack
Generate a configuration file with fluentd -s conf, you can specify the path to your configuration file. Edit the configuration to define a source as being from your CloudStack host. For instance if you a running a development environment locally:
<source>
@type cloudstack
host localhost
apikey $cloudstack_apikey
secretkey $cloudstack_secretkey
# optional
protocol http # https or http, default https
path /client/api # default /client/api
port 8080 # default 443
#interval 300 # min 300, default 300
ssl false # true or false, default true
domain_id $cloudstack_domain_id
tag cloudstack
</source>
There is currently a small bug in the interval definition so I commented it out. You also want to define the tag explicitly as being cloudstack. You can then create a <match> section in the configuration file. To keep it simple at first, we will simply echo the events to stdout, therefore just add:
<match cloudstack.**>
@type stdout
</match>
Run fluentd with fluentd -c conf/fluent.conf &, browse the CloudStack UI, create a VM, create a service offering. Once the interval is passed you will see the events being written to stdout:
I cut some of the output for brevity, note that I do have an interval listed as 3 because I did not want to wait 300 minutes. Therefore I installed from source and patched the plugin, it should be fixed in the source soon. You might have a different endpoint and of course different keys, and don't worry about me sharing that secret_key I am using a simulator, that key is already gone.
Plugging MongoDB
Getting the events and usage information on stdout is interesting, but the kicker comes from storing the data in a database or a search index. In this section we show to get closer to reality and use MongoDB to store the data. MongoDB is an open source document database which is schemaless and stores document in JSON format (BSON actually). Installation and query syntax of MongoDB is beyond the scope of this chapter. MongoDB clusters can be setup with replication and sharding, in this section we use MongoDB on a single host with no sharding or replication. To use MongoDB as a storage backend for the events, we first need to install mongodb. On single OSX node this is as simple as sudo port install mongodb. For other OS use the appropriate package manager. You can then start mongodb with sudo mongod --dbpath=/path/to/your/databases. Create a fluentd database and a fluentd user with read/write access to it. In the mongo shell do:
We then need to install the fluent-plugin-mongodb. Still using gem this will be done like so:
$sudo gem install fluent-plugin-mongo.
The complete documentation also explains how to modify the configuration of fluentd to use this backend. Previously we used stdout as the output backend, to use mongodb we just need to write a different <match> section like so:
# Single MongoDB
<match cloudstack.**>
@type mongo
host fluentd
port 27017
database fluentd
collection test
# for capped collection
capped
capped_size 1024m
# authentication
user fluentd
password foobar
# flush
flush_interval 10s
</match>
Note that you cannot have multiple match section for the same tag pattern.
To view the events/usages in Mongo, simply start a mongo shell with mongo -u fluentd -p foobar fluentd and list the collections. You will see the test collection:
$ mongo -u fluentd -p foobar fluentd
MongoDB shell version: 2.4.7
connecting to: fluentd
Server has startup warnings:
Fri Nov 1 13:11:44.855 [initandlisten]
Fri Nov 1 13:11:44.855 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
> show collections
system.indexes
system.users
test
Couple MongoDB commands will get your rolling, db.getCollection, count() and findOne():
