Plugin Development
To install a plugin, please put the ruby script in the /etc/fluent/plugin directory.
Alternatively, you can create a Ruby Gem package that includes a lib/fluent/plugin/<TYPE>_<NAME>.rb file. The TYPE is:
infor input pluginsoutfor output pluginsfilterfor filter pluginsparserfor parser pluginsformatterfor formatter pluginsstoragefor storage pluginsbuffor buffer pluginssdfor service discovery
For example, an email Output plugin would have the path: lib/fluent/plugin/out_mail.rb. The packaged gem can be distributed and installed using RubyGems. For further information, please see the list of Fluentd plugins for third-party plugins.
The following slides can help the user understand how Fluentd works before they dive into writing their own plugins.
(The slides are taken from Naotoshi Seo's RubyKaigi 2014 talk.)
This slide is based on Fluentd v0.12. There are many difference between v0.12 and v1 API, but it may help our understanding about Fluent's total design.
Fluentd now has two active versions, v1 and v0.12. v1 is current stable and v1 has brand-new Plugin API. v0.12 is old stable and v0.12 has old Plugin API.
The important point is v1 supports v1 and v0.12 APIs. It means the plugin for v0.12 works with v1.
We recommend to use new v1 plugin API for new plugins.
If you have a problem with existing plugins or new feature idea, sending a patch is better. If the plugin author is non-active, try to become new plugin maintainer first. Forking a plugin and release alternative plugin, e.g. fluent-plugin-xxx-alt, is final approach.
To create a plugin as a ruby script (to put it on /etc/fluent/plugin), just write a <TYPE>_<NAME>.rb file by editor, IDE or anything you prefer.
# in_my_awesome.rbrequire 'fluent/plugin/input'module Fluentmodule Pluginclass MyAwesomeInput < InputFluent::Plugin.register_input('my_awesome', self) # for "@type my_awesome" in configurationdef configure(conf)superenddef startsuper# ...endendendend
See each plugin development article, "How to write XXX plugin", for API details.
Single ruby script is easy to write, but hard to test, to manage versions and to publish it. If you want to publish a plugin under version control, you should use bundle gem to create the plugin source tree and init it as git repository (it requires bundler gem in your ruby environment): bundle gem fluent-plugin-my_awesome. It generates source code directory tree under lib, the simple fluent-plugin-my_awesome.gemspec file, README.md and some other files.
Fluentd plugin projects use a bit different code tree under lib from typical ruby projects. Take care about to keep lib/fluent/plugin/<TYPE>_<NAME>.rb paths.
Generate a project skeleton for creating a Fluentd plugin as a Gem package.
For example generate input http2 plugin project skeleton:
$ fluent-plugin-generate input http2License: Apache-2.0create Gemfilecreate README.mdcreate Rakefilecreate fluent-plugin-http2.gemspeccreate lib/fluent/plugin/in_http2.rbcreate test/helper.rbcreate test/plugin/test_in_http2.rbInitialized empty Git repository in /path/to/fluent-plugin-http2/.git/$ treefluent-plugin-http2/├── Gemfile├── LICENSE├── README.md├── Rakefile├── fluent-plugin-http2.gemspec├── lib│ └── fluent│ └── plugin│ └── in_http2.rb└── test├── helper.rb└── plugin└── test_in_http2.rb5 directories, 8 files
If you want to generate a project skeleton without LICENSE, use --no-license option. For more details, see fluent-plugin-generate --help.
Using fluent-plugin-generate command is good starting point to develop Fluentd plugins.
Run fluentd with the -vv option to show debug messages:
$ fluentd -vv
The stdout and copy output plugins are useful for debugging. The stdout output plugin dumps matched events to the console. It can be used as follows:
# You want to debug this plugin.<source>@type your_custom_input_plugin</source># Dump all events to stdout.<match **>@type stdout</match>
The copy output plugin copies matched events to multiple output plugins. You can use it in conjunction with the stdout plugin:
<source>@type forward</source># Use the forward Input plugin and the fluent-cat command to feed events:# $ echo '{"event":"message"}' | fluent-cat test.tag<match test.tag>@type copy# Dump the matched events.<store>@type stdout</store># Feed the dumped events to your plugin.<store>@type your_custom_output_plugin</store></match>
You can use stdout filter instead of copy and stdout combination. The result is same as above but more simpler.
<source>@type forward</source><filter>@type stdout</filter><match test.tag>@type your_custom_output_plugin</match>
Fluentd provides unit test frameworks for plugins:
Fluent::Test::Driver::InputTest driver for input plugins.Fluent::Test::Driver::OutputTest driver for output plugins.Fluent::Test::Driver::FilterTest driver for filter plugins
Fluentd core project strongly recommends to use test-unit as a unit test library. Fluentd's test drivers assume that the test code uses it. Add test-unit into the development dependency in your gemspec, add Rake task to run tests in your Rakefile and write test code in test/plugin/test_in_my_awesome.rb.
# in gemspecGem::Specification.new do |gem|gem.name = "fluent-plugin-my_awesome"# ...gem.add_runtime_dependency "fluentd"gem.add_development_dependency "test-unit"end# in Rakefilerequire 'rake/testtask'Rake::TestTask.new(:test) do |test|test.libs << 'lib' << 'test'test.pattern = 'test/**/test_*.rb'test.verbose = trueend
Then, run bundle exec rake test to run all tests in your test code.
See Writing Plugin Test Code for more details about writing tests.
You have a snippet of README.md if you generate project skeleton using fluent-plugin-generate.
For example:
# fluent-plugin-http2[Fluentd](https://fluentd.org/) input plugin to do something.TODO: write description for you plugin.## Installation### RubyGems```$ gem install fluent-plugin-http2```### BundlerAdd following line to your Gemfile:```rubygem "fluent-plugin-http2"```And then execute:```$ bundle```## ConfigurationYou can generate configuration template:```$ fluent-plugin-config-format input http2```You can copy and paste generated documents here.## Copyright* Copyright(c) 2017- John Doe* License* Apache License, Version 2.0
You should write plugin description and configurations.
You can generate documents for configuration using fluent-plugin-config-format command.
Example (input dummy):
$ fluent-plugin-config-format -c input dummy## Plugin helpers* thread* storage* See also: Fluent::Plugin::Input## Fluent::Plugin::DummyInput* **tag** (string) (required): The value is the tag assigned to the generated events.* **size** (integer) (optional): The number of events in event stream of each emits.* Default value: `1`.* **rate** (integer) (optional): It configures how many events to generate per second.* Default value: `1`.* **auto_increment_key** (string) (optional): If specified, each generated event has an auto-incremented key field.* **suspend** (bool) (optional): The boolean to suspend-and-resume incremental value after restart* **dummy** () (optional): The dummy data to be generated. An array of JSON hashes or a single JSON hash.* Default value: `[{"message"=>"dummy"}]`.
For more details, see fluent-plugin-config-format --help.
Slides: Dive into Fluentd Plugin
(outdated)
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
