exec_filter
The
out_exec_filter Buffered Output plugin 1) executes an external program using an event as input; and, 2) reads a new event from the program output.By default, it passes tab-separated values (TSV) to the standard input and reads TSV from the standard output.
It is included in Fluentd's core.
<match pattern>
@type exec_filter
command cmd arg arg
<format>
@type tsv
keys k1,k2,k3
</format>
<parse>
@type tsv
keys k1,k2,k3,k4
</parse>
<inject>
tag_key k1
time_key k2
time_format %Y-%m-%d %H:%M:%S
</inject>
</match>
Please see the Configuration File article for the basic structure and syntax of the configuration file.
When using the JSON format in
<parse> section, this plugin uses the Yajl library to parse the program output. Yajl buffers data internally so the output is not always instantaneous.- Synchronous
The value must be
exec_filter.type | default | version |
string | required parameter | 0.14.0 |
The command (program) to execute. The
out_exec_filter plugin passes the incoming event to the program input and receives the filtered event from the program output.type | default | version |
integer | 1 | 0.14.0 |
The number of spawned processes for
command.If the number is larger than 2, fluentd uses spawned processes by round robin fashion.
type | default | version |
string | nil | 0.14.0 |
Respawn command when the command exits. By default, it is disabled.
If you specify a positive number, it tries to respawn until specified times. If you specify
inf or -1, it tries to respawn forever.type | default | version |
string | nil | 0.14.0 |
The tag of the event.
type | default | version |
size | 10240 | 0.14.9 |
The default block size to read if parser requires partial read.
type | default | version |
time | 0 | 0.14.0 |
Suppress error logs during this interval.
By default, all the logs are emitted.
This parameter is deprecated. Use
<format> section.The format used to map the incoming event to the program input.
This parameter is deprecated. Use
<parse> section.The format used to process the program output.
The format used to map the incoming events to the program input.
type | default | version |
string | tsv | 0.14.9 |
Overwrites the default value in this plugin.
The format used to process the program output.
type | default | version |
string | tsv | 0.14.9 |
Overwrites the default value in this plugin.
type | default | version |
string | nil | 0.14.9 |
Overwrites the default value in this plugin.
type | default | version |
string | nil | 0.14.9 |
Overwrites the default value in this plugin.
type | default | version |
bool | true | 0.14.9 |
Overwrites the default value in this plugin.
type | default | version |
enum | float | 0.14.9 |
Overwrites the default value in this plugin.
type | default | version |
enum | float | 0.14.9 |
Overwrite default value in this plugin.
type | default | version |
enum | interval | 0.14.9 |
Overwrites the default value in this plugin.
type | default | version |
integer | 1 | 0.14.9 |
Overwrites the default value in this plugin.
Here is an example written in Ruby:
require 'json'
require 'msgpack'
begin
while line = STDIN.gets # continue to read a event from stdin
line.chomp!
# Input format depends on exec_filter's in_format setting
json = JSON.parse(line)
# main processing. You can do anything, mutate record, access to database and etc.
json['new_field'] = "Hey from exec_filter script!"
# Write data to stdout. Output format depends on exec_filter's out_format setting
STDOUT.print MessagePack.pack(json)
# Call flush to avoid buffering events
STDOUT.flush
end
rescue Interrupt # Ignore Interrupt exception because it happens during exec_filter shutdown
end
Corresponding configuration:
<match test.**>
@type exec_filter
command ruby /path/to/ruby_script.rb
tag filtered.exec
<format>
@type json
</format>
<parse>
@type msgpack
</parse>
<buffer>
flush_interval 10s
</buffer>
</match>
You may convert this script into your preferred language accordingly.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.
Last modified 2yr ago