How to Write Formatter Plugin

Search…
Fluentd supports pluggable, customizable formats for output plugins. The plugin filenames starting with formatter_ are registered as Formatter Plugins.
See Plugin Base Class API for more details on the common APIs of all the plugins.
Following is an example of a custom formatter (formatter_my_csv.rb) that outputs events in CSV format. It takes a required parameter called csv_fields and outputs the fields. It assumes that the values of the fields are valid CSV fields.
1
require 'fluent/plugin/formatter'
2
​
3
module Fluent::Plugin
4
  class MyCSVFormatter < Formatter
5
    # Register MyCSVFormatter as 'my_csv'.
6
    Fluent::Plugin.register_formatter('my_csv', self)
7
​
8
    config_param :csv_fields, :array, value_type: :string
9
​
10
    # This method does further processing. Configuration parameters can be
11
    # accessed either via `conf` hash or member variables.
12
    def configure(conf)
13
      super
14
    end
15
​
16
    # This is the method that formats the data output.
17
    def format(tag, time, record)
18
      values = []
19
​
20
      # Look up each required field and collect them from the record
21
      @csv_fields.each do |field|
22
        v = record[field]
23
        unless v
24
          log.error "#{field} is missing."
25
        end
26
        values << v.to_s
27
      end
28
​
29
      # Output by joining the fields with a comma
30
      values.join(',')
31
    end
32
  end
33
end
Copied!
Save this as formatter_my_csv.rb in a loadable plugin path.
With out_file output plugin:
1
# ...
2
​
3
<match test>
4
  @type file
5
  path /path/to/output/file
6
  <format>
7
    @type my_csv
8
    csv_fields k1,k2
9
  </format>
10
</match>
Copied!
For a matched record e.g. {"k1": 100, "k2": 200}, the output CSV file would look like this:
1
100,200
Copied!
How To Use Formatters From Plugins
Formatter plugins are designed to be used from other plugins, like Input, Filter and Output. Formatter plugin helper helps achieve this:
1
# in class definition
2
helpers :formatter
3
​
4
# in #configure
5
@formatter = formatter_create(type: 'json')
6
​
7
# in #filter, #format or ...
8
es.each do |time, record|
9
  row = @formatter.format(@tag, time, record)
10
  # ...
11
end
Copied!
See Formatter Plugin Helper API for details.
Methods
The formatter plugins implement filter method to format the input Hash record as a String object.
#format(tag, time, record)
It receives an event represented by tag, time and record; and, after formatting returns a String object.
Formatter plugins must implement this method.
Writing Tests
Fluentd formatter plugin has one or more points to be tested. Others (parsing configurations, controlling buffers, retries, flushes and many others) are controlled by the Fluentd core.
Fluentd also provides test driver for plugins. You can write tests for your own plugins very easily:
1
# test/plugin/test_formatter_your_own.rb
2
​
3
require 'test/unit'
4
require 'fluent/test/driver/formatter'
5
​
6
# your own plugin
7
require 'fluent/plugin/formatter_your_own'
8
​
9
class FormatterYourOwnTest < Test::Unit::TestCase
10
  def setup
11
    # common setup
12
  end
13
​
14
  CONFIG = %[
15
    fields a,b,c
16
  ]
17
​
18
  def create_driver(conf = CONF)
19
    Fluent::Test::Driver::Formatter.new(Fluent::Plugin::YourOwnFormatter).configure(conf)
20
  end
21
​
22
  sub_test_case 'configured with invalid configurations' do
23
    test 'empty' do
24
      assert_raise(Fluent::ConfigError) do
25
        create_driver('')
26
      end
27
    end
28
    # ...
29
  end
30
​
31
  sub_test_case 'plugin will format record' do
32
    test 'record has a field' do
33
      d = create_driver(CONFIG)
34
      tag = 'test'
35
      time = event_time
36
      record = { 'message' => 'This is message' }
37
      formatted = d.instance.format(tag, time, record)
38
      expected = '...'
39
      assert_equal(expected, formatted)
40
    end
41
  end
42
end
Copied!
Overview of Tests
Testing for formatter plugins is mainly for:
Validation of configuration parameters (i.e. #configure)
Validation of the formatted records
To make testing easy, the plugin test driver provides a logger and the functionality to override the system, parser and other configurations.
The lifecycle of the plugin and its test driver is:
1.
Instantiate plugin driver which then instantiates the plugin
2.
Configure plugin
3.
Run test code
4.
Assert results of tests by data provided by the driver
For:
configuration tests, repeat steps # 1-2
full feature tests, repeat step # 1-4
For more details, see Testing API for Plugins.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
How to Write Filter Plugin
How to Write Output Plugin
Last modified 4mo ago
Copy link
Contents
How To Use Formatters From Plugins
Methods
Writing Tests
Overview of Tests