regexpparser plugin parses logs by given regexp pattern. If the parameter value starts and ends with "/", it is considered to be a regexp. The regexp must have at least one named capture (?\PATTERN). If the regexp has a capture named
time, this is configurable, it is used as the time of the event. You can specify the time format using the time_format parameter.
format /.../ # regexp parser is used
format json # json parser is used
Specify the field for event time. Default is
Specify time format for
If you want to keep time field in the record, set
true. Default is
Although every parsed field has type
stringby default, you can specify other types. This is useful when filtering particular fields numerically or storing data with sensible type information.
The syntax is
As demonstrated above, "," is used to delimit field-type pairs while ":" is used to separate a field name with its intended type.
Unspecified fields are parsed at the default string type.
The list of supported types are shown below:
- integer ("int" would NOT work!)
arraytypes, there is an optional third field after the type name. For the "time" type, you can specify a time format like you would in
For the "array" type, the third field specifies the delimiter (the default is ","). For example, if a field called "item_ids" contains the value "3,4,5",
types item_ids:arrayparses it as ["3", "4", "5"]. Alternatively, if the value is "Adam|Alice|Bob",
types item_ids:array:|parses it as ["Adam", "Alice", "Bob"].
format /^\[(?<logtime>[^\]]*)\] (?<name>[^ ]*) (?<title>[^ ]*) (?<id>\d*)$/
time_format %Y-%m-%d %H:%M:%S %z
With this config:
[2013-02-28 12:00:00 +0900] alice engineer 1
This incoming log is parsed as:
1362020400 (2013-02-28 12:00:00 +0900)
"name" : "alice",
"id" : 1
fluentd-ui's in_tail editor helps your regexp testing. Another way, Fluentular is a great website to test your regexp for Fluentd configuration.
NOTE: You may hit Application Error at Fluentular due to heroku free plan limitation. Retry a few hours later or use fluentd-ui instead.
If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.