Versions | v0.14 (td-agent3) | v0.12 (td-agent2) | v0.10 (td-agent1)

This page is for v0.14, not the latest stable version which is v0.12. For the latest stable version of this article, click here.


Writing plugins

Table of Contents

Installing custom plugins

To install a plugin, please put the ruby script in the /etc/fluent/plugin directory.

Alternatively, you can create a Ruby Gem package that includes a lib/fluent/plugin/<TYPE>_<NAME>.rb file. The TYPE is:

  • in for input plugins
  • out for output plugins
  • filter for filter plugins
  • parser for parser plugins
  • formatter for formatter plugins
  • storage for storage plugins
  • buf for buffer plugins

For example, an email Output plugin would have the path: lib/fluent/plugin/out_mail.rb. The packaged gem can be distributed and installed using RubyGems. For further information, please see the list of Fluentd plugins for third-party plugins.

Overview

The following slides can help the user understand how Fluentd works before they dive into writing their own plugins.

(The slides are taken from Naotoshi Seo’s RubyKaigi 2014 talk.)

This slide is based on Fluentd v0.12. There are many difference between v0.12 and v0.14, but it may help our understanding about Fluent's total design.

Writing plugins

This section should be updated when Fluentd provides plugin generator.

To create a plugin as a ruby script (to put it on /etc/fluent/plugin), just write a <TYPE>_<NAME>.rb file by editor, IDE or anything you prefer.

# in_my_awesome.rb
require 'fluent/plugin/input'

module Fluent
  module Plugin
    class MyAwesomeInput < Input
      Fluent::Plugin.register_input('my_awesome', self) # for "@type my_awesome" in configuration

      def configure(conf)
        super
      end

      def start
        super
        # ...
      end
    end
  end
end

See the Plugin API details for what should be written / API details of each plugin types.

Single ruby script is easy to write, but hard to test, to manage versions and to publish it. If you want to publish a plugin uder version control, you should use bundle gem to create the plugin source tree and init it as git repository (it requires bundler gem in your ruby environment): bundle gem fluent-plugin-my_awesome. It generates source code directory tree under lib, the simple fluent-plugin-my_awesome.gemspec file, README.md and some other files.

Fluentd plugin projects use a bit different code tree under lib from typical ruby projects. Take care about to keep lib/fluent/plugin/<TYPE>_<NAME>.rb paths.

Debugging plugins

Run fluentd with the -vv option to show debug messages:

$ fluentd -vv

The stdout and copy output plugins are useful for debugging. The stdout output plugin dumps matched events to the console. It can be used as follows:

# You want to debug this plugin.
<source>
  @type your_custom_input_plugin
</source>

# Dump all events to stdout.
<match **>
  @type stdout
</match>

The copy output plugin copies matched events to multiple output plugins. You can use it in conjunction with the stdout plugin:

<source>
  @type forward
</source>

# Use the forward Input plugin and the fluent-cat command to feed events:
#  $ echo '{"event":"message"}' | fluent-cat test.tag
<match test.tag>
  @type copy

  # Dump the matched events.
  <store>
    @type stdout
  </store>

  # Feed the dumped events to your plugin.
  <store>
    @type your_custom_output_plugin
  </store>
</match>

You can use stdout filter instead of copy and stdout combination. The result is same as above but more simpler.

<source>
  @type forward
</source>

<filter>
  @type stdout
</filter>

<match test.tag>
  @type your_custom_output_plugin
</match>

Writing tests for plugins

Fluentd provides unit test frameworks for plugins:

Fluent::Test::Driver::Input
  Test driver for input plugins.

Fluent::Test::Driver::Output
  Test driver for output plugins.

Fluent::Test::Driver::Filter
  Test driver for filter plugins

Fluentd core project strongly recommends to use test-unit as a unit test library. Fluentd’s test drivers assume that the test code uses it. Add test-unit into the development dependency in your gemspec, add Rake task to run tests in your Rakefile and write test code in test/plugin/test_in_my_awesome.rb.

# in gemspec
Gem::Specification.new do |gem|
  gem.name = "fluent-plugin-my_awesome"
  # ...
  gem.add_runtime_dependency     "fluentd"
  gem.add_development_dependency "test-unit"
end

# in Rakefile
require 'rake/testtask'
  Rake::TestTask.new(:test) do |test|
  test.libs << 'lib' << 'test'
  test.pattern = 'test/**/test_*.rb'
  test.verbose = true
end

Then, run bundle exec rake test to run all tests in your test code.

See Writing Plugin Test Code for more details about writing tests.

Further Reading

Last updated: 2017-02-14 11:01:37 UTC

Versions | v0.14 (td-agent3) | v0.12 (td-agent2) | v0.10 (td-agent1)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF), originally invented by Treasure Data, Inc. All components are available under the Apache 2 License.

Interested in the Fluentd Newsletters?