Plugin Development

Search…
Plugin Development
Installing Custom Plugins
To install a plugin, put the Ruby script in /etc/fluent/plugin directory.
Alternatively, you can create a Ruby Gem package that includes:
1
lib/fluent/plugin/<TYPE>_<NAME>.rb
Copied!
The TYPE prefix could be:
prefix
plugin type
in
input
out
output
filter
filter
parser
parser
formatter
formatter
storage
storage
buf
buffer
sd
service discovery
For example, an email output plugin would have the path:
1
lib/fluent/plugin/out_mail.rb
Copied!
The packaged gem can be distributed and installed using RubyGems. For further information, see the list of Fluentd plugins for third-party plugins.
Overview
The following slides can help understand how Fluentd works before diving into writing custom plugins.
These slides are from Naotoshi Seo's RubyKaigi 2014 talk.
The slides are based on Fluentd v0.12. There are many differences between v0.12 and v1 API, but it may help build an understanding of overall Fluentd's design.
Fluentd Version and Plugin API
Fluentd now has two active versions, v1 and v0.12. v1 is the current stable with the brand-new Plugin API. v0.12 is the old stable and it has the old Plugin API.
The important point is v1 supports v1 and v0.12 APIs. It means the plugin for v0.12 works with v1.
It is recommended to use the new v1 plugin API for writing new plugins.
Send a patch or fork?
If you have a problem with any existing plugins or a new feature idea, sending a patch is better. If the plugin author is non-active, try to become its new plugin maintainer first. Forking a plugin and release its alternative version, e.g. fluent-plugin-xxx-alt is considered the last option.
Be careful not breaking Fluentd's core functionality
Any fluentd plugin can unknowingly break fluentd completely (and possibly break other plugins) by requiring some incompatible modules.
One typical example is reloading yajl/json_gem from plugin. (yajl/json_gem compatibility layer is problematic, not for yajl itself) It breaks a functionality to parse Fluentd's configuration completely.
Fluentd's plug-in mechanism has a merit to extend functionality, but plugin developer must be careful a possibility of breaking it.
Writing Plugins
To create a plugin as a Ruby script (to put it in /etc/fluent/plugin/ directory), create a <TYPE>_<NAME>.rb file in any editor/IDE of your choice.
Example:
1
# in_my_awesome.rb
2
require 'fluent/plugin/input'
3
​
4
module Fluent
5
  module Plugin
6
    class MyAwesomeInput < Input
7
      # For `@type my_awesome` in configuration file
8
      Fluent::Plugin.register_input('my_awesome', self)
9
​
10
      def configure(conf)
11
        super
12
      end
13
​
14
      def start
15
        super
16
        # ...
17
      end
18
    end
19
  end
20
end
Copied!
See the respective plugin development article, "How to write XXX plugin", for API details.
A single Ruby script is easy to write but hard to test, manage, version. publish, etc.
If you want to publish a plugin under version control, you should use bundle gem to create the plugin source tree and then git init it. It requires the bundler gem in your Ruby environment.
Example:
1
$ bundle gem fluent-plugin-my_awesome
Copied!
It generates source code directory tree under lib, fluent-plugin-my_awesome.gemspec file, README.md and other relevant files.
Fluentd plugin projects use a bit different code tree under lib i.e.:
1
lib/fluent/plugin/<TYPE>_<NAME>.rb
Copied!
Generating Plugin Project Skeleton
To make things easier, fluent-plugin-generate is provided to generate the project skeleton for writing a Fluentd plugin as a Gem package.
For example, to generate the project skeleton for an input plugin http2, the command would be:
1
$ fluent-plugin-generate input http2
2
License: Apache-2.0
3
        create Gemfile
4
        create README.md
5
        create Rakefile
6
        create fluent-plugin-http2.gemspec
7
        create lib/fluent/plugin/in_http2.rb
8
        create test/helper.rb
9
        create test/plugin/test_in_http2.rb
10
Initialized empty Git repository in /path/to/fluent-plugin-http2/.git/
11
$ tree
12
fluent-plugin-http2/
13
├── Gemfile
14
├── LICENSE
15
├── README.md
16
├── Rakefile
17
├── fluent-plugin-http2.gemspec
18
├── lib
19
│   └── fluent
20
│       └── plugin
21
│           └── in_http2.rb
22
└── test
23
    ├── helper.rb
24
    └── plugin
25
        └── test_in_http2.rb
26
​
27
5 directories, 8 files
Copied!
If you want to generate a project skeleton without LICENSE, use --no-license option. For more details, see fluent-plugin-generate --help.
Debugging Plugins
Run fluentd with the -vv command-line option to show debugging messages:
1
$ fluentd -vv
Copied!
The stdout and copy output plugins are useful for debugging. The stdout output plugin dumps matched events to the standard output (console). It can be configured like this:
1
# You want to debug this plugin.
2
<source>
3
  @type your_custom_input_plugin
4
</source>
5
​
6
# Dump all events to stdout.
7
<match **>
8
  @type stdout
9
</match>
Copied!
The copy output plugin copies the matched events to the multiple output plugins. You can use it in conjunction with the stdout plugin:
1
<source>
2
  @type forward
3
</source>
4
​
5
# Use the forward Input plugin and the fluent-cat command to feed events:
6
#  $ echo '{"event":"message"}' | fluent-cat test.tag
7
<match test.tag>
8
  @type copy
9
​
10
  # Dump the matched events.
11
  <store>
12
    @type stdout
13
  </store>
14
​
15
  # Feed the dumped events to your plugin.
16
  <store>
17
    @type your_custom_output_plugin
18
  </store>
19
</match>
Copied!
You can use stdout filter instead of copy and stdout combination. The result is the same as above but more simpler.
1
<source>
2
  @type forward
3
</source>
4
​
5
<filter>
6
  @type stdout
7
</filter>
8
​
9
<match test.tag>
10
  @type your_custom_output_plugin
11
</match>
Copied!
Writing Tests for Plugins
Fluentd provides unit test frameworks for plugins:
1
Fluent::Test::Driver::Input   # Test driver for input plugins.
2
Fluent::Test::Driver::Output  # Test driver for output plugins.
3
Fluent::Test::Driver::Filter  # Test driver for filter plugins.
Copied!
It is strongly recommended to use test-unit as the unit test library. Fluentd's test drivers assume that the test code uses it. Add test-unitas the development dependency in your gemspec, add Rake task to run tests in your Rakefile and write test code in test/plugin/test_in_my_awesome.rb.
gemspec
1
Gem::Specification.new do |gem|
2
  gem.name = 'fluent-plugin-my_awesome'
3
  # ...
4
  gem.add_runtime_dependency     'fluentd'
5
  gem.add_development_dependency 'test-unit'
6
end
Copied!
Rakefile
1
require 'rake/testtask'
2
  Rake::TestTask.new(:test) do |test|
3
  test.libs << 'lib' << 'test'
4
  test.pattern = 'test/**/test_*.rb'
5
  test.verbose = true
6
end
Copied!
Run bundle to execute tests:
1
$ bundle exec rake test
Copied!
See Writing Plugin Test Code for more details on writing tests.
Writing Documents for Plugins
Following is a snippet of README.md showing the project skeleton generated with fluent-plugin-generate.
For example:
1
# fluent-plugin-http2
2
​
3
[Fluentd](https://fluentd.org/) input plugin to do something.
4
​
5
TODO: write description for you plugin.
6
​
7
## Installation
8
​
9
### RubyGems
10
​
11
```
12
$ gem install fluent-plugin-http2
13
```
14
​
15
### Bundler
16
​
17
Add following line to your Gemfile:
18
​
19
```ruby
20
gem "fluent-plugin-http2"
21
```
22
​
23
And then execute:
24
​
25
```
26
$ bundle
27
```
28
​
29
## Configuration
30
​
31
You can generate configuration template:
32
​
33
```
34
$ fluent-plugin-config-format input http2
35
```
36
​
37
You can copy and paste generated documents here.
38
​
39
## Copyright
40
​
41
* Copyright(c) 2017- John Doe
42
* License
43
  * Apache License, Version 2.0
Copied!
You should write the plugin description and configurations.
You can generate configuration using fluent-plugin-config-format command.
Example (input dummy):
1
$ fluent-plugin-config-format -c input dummy
2
## Plugin helpers
3
​
4
* thread
5
* storage
6
​
7
* See also: Fluent::Plugin::Input
8
​
9
## Fluent::Plugin::DummyInput
10
​
11
* **tag** (string) (required): The value is the tag assigned to the generated events.
12
* **size** (integer) (optional): The number of events in event stream of each emits.
13
  * Default value: `1`.
14
* **rate** (integer) (optional): It configures how many events to generate per second.
15
  * Default value: `1`.
16
* **auto_increment_key** (string) (optional): If specified, each generated event has an auto-incremented key field.
17
* **suspend** (bool) (optional): The boolean to suspend-and-resume incremental value after restart
18
* **dummy** () (optional): The dummy data to be generated. An array of JSON hashes or a single JSON hash.
19
  * Default value: `[{"message"=>"dummy"}]`.
Copied!
For more details, see fluent-plugin-config-format --help.
Further Reading
​Slides: Fluentd v0.14 Plugin API Details​
​Slides: Dive into Fluentd Plugin (outdated)
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Scala
How to Write Input Plugin
Last modified 6mo ago
Copy link
Contents
Installing Custom Plugins
Overview
Fluentd Version and Plugin API
Send a patch or fork?
Be careful not breaking Fluentd's core functionality
Writing Plugins
Generating Plugin Project Skeleton
Debugging Plugins
Writing Tests for Plugins
Writing Documents for Plugins