How to Write Base Plugin

Search…
All plugin types are subclasses of Fluent::Plugin::Base in Fluentd v1 or later. The Base class has some features and methods that provide the basic mechanism as plugins. This page shows these methods provided by Fluent::Plugin::Base, and other methods provided commonly in some type of plugins.
The methods listed below are considered as public methods, and will be maintained not to break compatibility. Other methods may be changed without compatibility consideration.
1
require 'fluent/plugin/input' # This may be input, filter, output, parser, formatter, storage or buffer.
2
​
3
module Fluent::Plugin
4
  class MyExampleInput < Input
5
    # Plugins should be registered by calling `Fluent::Plugin.register_TYPE` method with name and `self`.
6
    # The first argument String is to identify the plugin in the configuration file.
7
    # The second argument is class of the plugin itself, `self` in most cases.
8
    Fluent::Plugin.register_input('my_example', self)
9
​
10
    desc 'The port number'
11
    # `config_param` Defines a parameter. You can refer the following parameter via @port instance variable.
12
    # Without `:default`, a parameter is required.
13
    config_param :port, :integer
14
​
15
    config_section :user, param_name: :users, multi: true, required: false do
16
      desc 'Username for authentication'
17
      config_param :username, :string
18
      desc 'Password for authentication'
19
      config_param :password, :string, secret: true
20
    end
21
​
22
    def configure(conf)
23
      super
24
​
25
      # If the configuration is invalid, raise `Fluent::ConfigError`.
26
      if @port <= 1024
27
        raise Fluent::ConfigError, "port number is too small: #{@port}"
28
      end
29
​
30
      @users.each do |user|
31
        if user.password.length < 5
32
          raise Fluent::ConfigError, "password is too short for user '#{user.username}'"
33
        end
34
      end
35
    end
36
​
37
    def start
38
      super
39
      # ...
40
    end
41
​
42
    def shutdown
43
      # ...
44
      super
45
    end
46
  end
47
end
Copied!
Class Methods
Base class provides methods to create configurable parameters for plugins. It also supports methods to provide system configurations to plugins.
.config_param(name, type = nil, **options, &block)
Defines a parameter.
name: the parameter name as a symbol
type: the parameter type
options: the options for the parameter
block: if given, convert the value via the given block
For details on types and options, see Types of Configuration Parameters.
Code Example:
1
config_param :name, :string, default: 'John Doe', alias: :full_name
2
​
3
config_param :pattern do |value|
4
  Regexp.compile(value)
5
end
6
​
7
config_param :delimiter, default: "\t" do |value|
8
  case value
9
  when /SPACE/i then ' '
10
  when /COMMA/i then ','
11
  else "\t"
12
  end
13
end
Copied!
.config_set_default(name, default_value)
Sets the default value of the parameter specified by name. If the default value already exists, it raises ArgumentError.
name: The name of the parameter.
default_value: Default value of the parameter.
Code Example:
1
# lib/fluent/plugin/buffer.rb
2
config_param :chunk_limit_size, :size, default: DEFAULT_CHUNK_LIMIT_SIZE
3
​
4
# lib/fluent/plugin/buf_file.rb
5
config_set_default :chunk_limit_size, DEFAULT_CHUNK_LIMIT_SIZE
Copied!
.config_set_desc(name, description)
Sets description of the parameter specified by name. If the description already exists, it raises ArgumentError. For internal use only! Use desc instead.
name: The name of the parameter.
description: Description of the parameter.
.desc(description)
Sets description of the immediately following parameter.
description: Description of the parameter.
Code Example:
1
desc 'The username'
2
config_param :user, :string
Copied!
.config_section(name, **options, &block)
Defines a section to construct structured (nested) configuration.
name: The name of the section.
options:
root: If true, this section is the root section. For internal use only!
param_name: The section name.
final: If true, the subclass of this class cannot override this section.
For example, the third0party plugins cannot override the buffer section.
init: If true, the parameters in this section must have default values.
Not applicable to third-party plugins.
required: If true, the section is required. Fluentd will raise
Fluent::ConfigError if the section is missing.
multi: If true, users can configure this section multiple times.
alias: Alias for this section.
Code Example:
1
config_section :user, multi: true do
2
  config_param :name, :string
3
  config_param :password, :string, secret: true
4
end
5
​
6
# nested sections
7
config_section :child, :param_name: 'children', required: false, multi: true do
8
  config_param :name, :string
9
  config_param :power, :integer, default: nil
10
  config_section :item, multi: true do
11
    config_param :name, :string
12
    config_param :flavor, :string
13
  end
14
end
Copied!
Configuration Example:
1
<user>
2
  name Alice
3
  password very-secret-password
4
</user>
5
​
6
# nested sections
7
<child>
8
  name Bob
9
  power 1000
10
  <item>
11
    name potion
12
    flavor very sour orange
13
  </item>
14
  <item>
15
    name gumi
16
    flavor super delicious apple
17
  </item>
18
</child>
Copied!
.configured_in(section_name)
Inherits section section_name defined in the super class.
section_name: The section name as Symbol.
.system_config
Returns Fluent::SystemConfig instance.
For more details, see System Configuration.
Code Example:
1
def configure(conf)
2
  super
3
​
4
  root_dir = system_config.root_dir
5
end
Copied!
.system_config_override(options = {})
Overrides the system configuration.
This is for internal use and plugin testing.
For more details, see System Configuration.
options: The system configuration as Hash.
Code Example:
1
test 'plugin instance can overwrite system_config if needed' do
2
  plugin = ExampleInput.new
3
  plugin.configure(conf)
4
  # ...
5
  plugin.system_config_override('workers' => 3)
6
  # ...
7
end
Copied!
Instance Methods
#initialize
Initializes the internal states (instance variables).
Call super if the plugin overrides this method.
Code Example:
1
def initialize
2
  super
3
​
4
  @internal_counter = 0
5
  @in_memory_cache = nil
6
end
Copied!
#configure(conf)
The conf parameter is an instance of Fluent::Config::Element. Call super if the plugin overrides this method. Fluentd's Configurable module (included in Base class) will traverse conf object, and set values from configurations or default values into the instance variables in super. So, the instance variables or accessor methods are available after super in #configure method.
The code to configure the plugin should be after super.
Code Example:
1
def configure(conf)
2
  super
3
​
4
  # cache_default_value is created/configured by config_param
5
  @in_memory_cache = Hash.new(@cache_default_value)
6
end
Copied!
NOTE: The return value of this method will be ignored.
#log
Returns Fluent::Log instance.
Log levels:
trace
debug
info
warn
error
fatal
For more details on Fluentd's logging mechanism, see Logging.
Code Example:
1
def start
2
  # ...
3
  log.info('This is info level log')
4
​
5
  # Evaluate only if log level is trace
6
  log.trace do
7
    # Heavy calculation to trace plugin behavior
8
    'This is trace level log'
9
  end
10
end
Copied!
#has_router?
Indicates whether the #router method exists or not. (default: false)
If the plugin uses event_emitter plugin helper, this method will return true.
NOTE: Input plugin enables event_emitter by default.
#start
This method is automatically called when Fluentd starts after the configuration. Call super if the plugin overrides this method.
Creating/opening timers, threads, listening sockets, file handles and others should be done in this method after super. Many of these may be provided as plugin helpers. See API details of each plugin helper.
Code Example:
1
def start
2
  super
3
​
4
  timer_execute(:my_example_timer, 30) do
5
    # Code that will be executed every 30 seconds
6
  end
7
end
Copied!
#stop
This method is automatically called first in the shutdown sequence of Fluentd. It should be used to manipulate flags to stop loops, e.g. network servers, gracefully. This method SHOULD NOT do anything that may raise errors.
Call super if the plugin overrides this method.
Code Example:
1
def start
2
  super
3
​
4
  @my_thread_running = true
5
  @my_thread = thread_create(:example_code) do
6
    if @my_thread_running
7
      log.debug 'loop is running'
8
    end
9
  end
10
end
11
​
12
def stop
13
  @my_thread_running = false
14
​
15
  super
16
end
Copied!
super should be called at last in methods of shutdown sequence: stop, before_shutdown, shutdown, after_shutdown, close and terminate.
#before_shutdown
This method is automatically called after #stop and before #shutdown. It may be used to control the flushing of buffered events in the shutdown sequence.
Call super if the plugin overrides this method. The third-party plugins do not need to implement this method in most cases.
#shutdown
This method is automatically called while shutting down. It may be used to close file handles, network connections, listening servers, and other resources that need cleanup. The event can be emitted in this method but not after this method is called.
Call super if the plugin overrides this method.
Code Example:
1
def shutdown
2
  @server.close
3
  records = my_convert_method(@server.rest_data)
4
  records.each do |record|
5
    router.emit(@tag, Fluent::Engine.now, record)
6
  end
7
​
8
  super
9
end
Copied!
#after_shutdown
This method is automatically called after #shutdown. It is used to control the emitting of events in shutdown sequence.
Call super if the plugin overrides this method. Third-party plugins do not need to implement this method in most cases.
#close
This method may be used to close those resources that cannot be closed in #shutdown.
Call super if the plugin overrides this method.
#terminate
This method may be used to re-initialize the internal states for the reuse of plugin instances in tests, etc.
Call super if the plugin overrides this method.
Methods for Input/Filter/Output
Following methods are available in the subclass of Input, Filter and Output:
.helpers(*symbols)
Includes the features of the plugin helpers.
Code Example:
1
module Fluent::Plugin
2
  class MyPlugin < Input
3
    Fluent::Plugin.register_input('my_plugin', self)
4
​
5
    # This call enables Timer and Storage plugin helpers
6
    helpers :timer, :storage
7
​
8
    # ...
9
  end
10
end
Copied!
It is strongly recommended to call this method at the top of the plugin class definition (just after calling #register_) to show what plugin helpers this plugin uses explicitly.
#plugin_id
Provides a unique ID string for the plugin instance. It might be specified by users in the configuration files, or generated automatically. The plugin must not expect any fixed formats for its return value.
#plugin_id_configured?
Indicates whether #plugin_id is configured by users or not.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
How to Write Input Plugin
How to Write Buffer Plugin
Last modified 4mo ago
Class Methods

`.config_param(name, type = nil, **options, &block)`

`.config_set_default(name, default_value)`

`.config_set_desc(name, description)`

`.desc(description)`

`.config_section(name, **options, &block)`

`.configured_in(section_name)`

`.system_config`

`.system_config_override(options = {})`

Instance Methods

`#initialize`

`#configure(conf)`

`#log`

`#has_router?`

`#start`

`#stop`

`#before_shutdown`

`#shutdown`

`#after_shutdown`

`#close`

`#terminate`

Methods for Input/Filter/Output

`.helpers(*symbols)`

`#plugin_id`

`#plugin_id_configured?`
