Versions | v0.14 (td-agent3)

This page is for v0.14, not the latest stable version which is v0.12. For the latest stable version of this article, click here.


Updating plugin for v0.14 from v0.12

This guide is for plugin authors to show how to update input/output/filter plugins written for Fluentd v0.12 or earlier.

There are something to be considered (see following “Updating Plugins Overview” section for details):

  • Plugins which uses v0.12 API will be supported between Fluentd v0.14 and v1 (will be obsoleted at v2).
  • Users can use new features of Fluentd v0.14 only with plugins using v0.14 API.
  • Plugins which uses v0.14 API don’t work on Fluentd v0.12.x.

Fluentd core team strongly recommend to use v0.14 API to make your plugins stable, consistent and easy to test.

Table of Contents

Updating Plugins Overview

These are steps to update your plugins safely.

  1. release a latest version for Fluentd v0.12.x
  2. update dependency
  3. update code and tests
  4. upate CI environments
  5. release the newer version for Fluentd v0.14.x and later

1. release a latest version

At first, you should make a git branch named as fluentd-v0.12 (if you are using git for that plugin), and release a latest patch version from that branch without any changes, except for fixing dependency about Fluentd ~> 0.12.0. This makes you possible to fix bugs and release newer versions for Fluentd v0.12 users without breaking anything.

  • make a branch for Fluentd v0.12 versions
  • fix dependency about Fluentd to ~> 0.12.0 (or later: ~> 0.12.26)
  • bump your gem’s version up to next patch version (for example: 0.4.1 -> 0.4.2)
  • release it to rubygems.org

2. update dependency

Following updates are on master branch. You should update dependency in gemspec at first to depend on Fluentd v0.14.

  • fix dependency about Fluentd to >= 0.14.0
  • execute bundle install

Is it correct to use >= 0.14.0 instead of ~> 0.14.0? Yes. Fluentd v1 will be compatible with v0.14.0 (especially for plugin APIs), so you should use >= 0.14.0. Let’s think about v2 few years later.

3. update code and tests

There are many difference between plugin types about updating code and tests. See “Updating Plugin Code” section below for each types of plugins.

  • update code and tests
  • confirm to run bundle exec rake test

4. update CI environments

If you have CI configurations like .travis.yml and appvayor.yml, these should be updated to support Fluentd v0.14. Fluentd v0.14 supports Ruby 2.1 or later. CI environments should not include Ruby 2.0 or earlier. It’s good idea to add latest Ruby (2.3 at Jun 2016).

  • remove Ruby 2.0 or ealier from CI environments
  • add Ruby 2.3 (or other latest version) to CI environments

5. release new version

This is last step. The new version should be major or minor version up, not patch version up. If the current major version of your gem is equal or larger than 1, you should bump major version up (e.g., from 1 to 2). If the current major version is 0, you should bump minor version up (e.g., from 0.4.2 to 0.5.0). Then, you can publish a new release which is available with Fluentd v0.14.

  • bump the version up
  • release it to rubygems.org

Updating Plugin Code

For all types of plugins, take care about these things:

  • require files which contains definitions of classes/modules referred in your plugin code
  • call super in #initialize, #configure, #start and #shutdown
  • use router.emit to emit events into Fluentd instead of Engine.emit

About updating tests, see “Test code” section for all plugin types.

Input plugins

For input plugins, points to be fixed are:

  • require “fluent/plugin/input” instead of “fluent/input”
  • fix superclass from Fluent::Input to Fluent::Plugin::Input
  • use compat_parameters plugin helper to keep compatibility of configurations to v0.12 style
  • use Fluent::Engine.now or Fluent::EventTime.now to create current time object instead of Time.now.to_i
  • update test code

Plugins will work well only with changes above.

Moreover, most input plugins create threads, timers, network serevers and/or parsers. It’s better to use plugin helpers to simplify code and to make tests stable. For more details, see Plugin Helper Overview.

Filter plugins

For filter plugins, points to be fixed are:

  • require “fluent/plugin/filter” instead of “fluent/filter”
  • fix superclass from Fluent::Filter to Fluent::Plugin::Filter
  • use compat_parameters plugin helper to keep compatibility of configurations to v0.12 style
  • update test code

Plugins will work well only with changes above. But if your plugin implements #filter_stream, remove it if possible. Overriding #filter_stream make it impossible to optimize filters' performance.

Moreover, many filter plugins uses parsers or formatters. It’s better to use plugin helpers for them to simplify code and make it easy to understand the way to configure the plugin.

Non-buffered output plugins

For output plugins (subclass of Fluent::Output), points to be fixed are:

  • require “fluent/plugin/output” instead of “fluent/output”
  • fix superclass from Fluent::Output to Fluent::Plugin::Output
  • use compat_parameters plugin helper to keep compatibility of configurations to v0.12 style
  • remove #emit method and implement #process(tag, es) method
  • update test code

If your output plugin emits events into Fluentd, follow these points too:

  • use event_emitter plugin helper to introduce router
  • use Fluent::Engine.now or Fluent::EventTime.now to create current time object instead of Time.now.to_i

It’s recommended to use plugin helpers if your plugin creates any of thread, timer socket, child process and/or parsers/formatters. It’s better to use plugin helpers to simplify code and to make tests stable. For more details, see Plugin Helper Overview.

TODO: add example of before / after

Buffered output plugins

For buffered output plugins (subclass of Fluent::BufferedOutput), points to be fixed are:

  • require “fluent/plugin/output” instead of “fluent/output”
  • fix superclass from Fluent::BufferedOutput to Fluent::Plugin::Output
  • use compat_parameters plugin helper to keep compatibility of configurations to v0.12 style
  • implement #compat_parameters_default_chunk_key to return empty string to show chunk key is not specified
  • fix config_set_default and its parameter names to override parameters in <buffer> section
  • remove #format_stream method if it is implemented in your plugin (it is not supported)
  • update test code

It’s recommended to use plugin helpers if your plugin creates any of thread, timer socket, child process and/or parsers/formatters. It’s better to use plugin helpers to simplify code and to make tests stable. For more details, see Plugin Helper Overview.

TODO: add example of before / after

ObjectBuffered output plugins

For object buffered output plugins (subclass of Fluent::ObjectBufferedOutput), points to be fixed are:

  • require “fluent/plugin/output” instead of “fluent/output”
  • fix superclass from Fluent::ObjectBufferedOutput to Fluent::Plugin::Output
  • use compat_parameters plugin helper to keep compatibility of configurations to v0.12 style
  • implement #compat_parameters_default_chunk_key to return "tag" to show chunk key is tag (or something else if your plugin overwrites #emit to change key)
  • fix config_set_default and its parameter names to override parameters in <buffer> section
  • fix #write method code not to use chunk.key, to use chunk.metadata.tag and #extract_placeholders
  • update test code

It’s recommended to use plugin helpers if your plugin creates any of thread, timer socket, child process and/or parsers/formatters. It’s better to use plugin helpers to simplify code and to make tests stable. For more details, see Plugin Helper Overview.

TODO: add example of before / after

TimeSliced output plugins

For time sliced output plugins (subclass of Fluent::TimeSlicedOutput), points to be fixed are:

  • require “fluent/plugin/output” instead of “fluent/output”
  • fix superclass from Fluent::TimeSlicedOutput to Fluent::Plugin::Output
  • use compat_parameters plugin helper to keep compatibility of configurations to v0.12 style
  • implement #compat_parameters_default_chunk_key to return "time" to show chunk key is time
  • set default value of timekey in <buffer> section if your plugin specifies default time_slice_format
  • fix config_set_default and its parameter names to override parameters in <buffer> section
  • fix #write method code not to use chunk.key, to use chunk.metadata.timekey and #extract_placeholders
  • update test code

It’s recommended to use plugin helpers if your plugin creates any of thread, timer socket, child process and/or parsers/formatters. It’s better to use plugin helpers to simplify code and to make tests stable. For more details, see Plugin Helper Overview.

TODO: add example of before / after

Multi output plugins

For multi output plugins (subclass of Fluent::MultiOutput), there are many points to be considered.

If the plugin uses <store> sections and instantiates plugins per each store section, use Fluent::Plugin::MultiOutput. See code to know how to use it: lib/fluent/plugin/multi_output.rb or some built-in plugins like out_copy, out_roundrobin.

Otherwise, your plugin does something curious for Fluentd. Read code of lib/fluent/plugin/output.rb and lib/fluent/plugin/bare_output.rb, and consider which is better for your plugin. But it is strongly unrecommended to use Fluent::Plugin::BareOutput for most use cases.

Output plugins using mixins

Fluent::HandleTagAndTimeMixin, Fluent::SetTagKeyMixin, Fluent::SetTimeKeyMixin

Use inject and compat_parameters plugin helper in plugin code.

Old configuration will be converted to new style configuration automatically if plugin code uses proper plugin helpers. So plugin users will not need to rewrite configuration immediately.

Fluentd shows converted new style configuration in startup log if user provides old style configuration. User can rewrite configuration refer to the log.

Before:

<match **>
  @type some_output
  include_tag_key true
  tag_key tag
  include_time_key true
  time_key time
  time_format %Y-%m-%d
</match>

After:

<match **>
  @type some_output
  <inject>
    tag_key tag
    time_key time
    time_format %Y-%m-%d
  </inject>
</match>

Fluent::HandleTagNameMixin

Related configurations:

  • remove_tag_prefix
  • remove_tag_suffix
  • add_tag_prefix
  • add_tag_suffix

Use extract_placeholders(template, chunk.metadata) in plugin code.

Use placeholders ${tag}, ${tag[0]}, ${tag[1]} in configuration.

Before:

<match input.access>
  @type some_output
  remove_tag_prefix input.
  tag               some.${tag}
  <record>
    ...
  </record>
</match>

After:

<match input.access>
  @type some_output
  tag some.${tag[1]}
  <record>
    ...
  </record>
</match>

Parser plugins

TODO: write

Formatter plugins

TODO: write

Test code

TODO: write

Last updated: 2017-02-02 01:40:30 UTC

Versions | v0.14 (td-agent3)

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.

Interested in the Fluentd Newsletters?