fluent-plugin-rewrite, a plugin for Fluentd
Requirements
fluent-plugin-rewrite | fluentd | ruby |
---|---|---|
>= 0.1.0 | >= v0.14.8 | >= 2.1 |
< 0.1.0 | >= v0.12.0 | >= 1.9 |
Component
RewriteOutput
Output plugin to rewrite messages' tags/values along with pattern matching and re-emit them.
Synopsis
<match apache.log.**>
@type rewrite
remove_prefix apache.log
add_prefix filtered
<rule>
key path
pattern \\?.+$
replace
</rule>
<rule>
key path
pattern (/[^/]+)\\?([^=]+)=(\\d)
replace \\1/\\2/\\3
</rule>
<rule>
key status
pattern ^500$
ignore true
</rule>
<rule>
key path
pattern ^\/(users|entries)
append_to_tag true
fallback others
</rule>
<rule>
key is_loggged_in
pattern 1
append_to_tag true
tag user
</rule>
</match>
Configuration
remove_prefix / add_prefix
remove_prefix apache.log
add_prefix filtered
- remove_prefix: removes the string from a prefix of tag.
- add_prefix: prepend the string to a tag.
rule: replace
For example, if you want to filter out query string form URL string:
<rule>
key path
pattern \\?.+$
replace
</rule>
It executes pattern matching against a value related with key
and replaces it with empty string if it matches.
/foo?bar=baz -> /foo
This time, if you want to rewrite path string along with some pattern:
<rule>
key path
pattern (/[^/]+)\\?([^=]+)=(\\d)
replace \\1/\\2/\\3
</rule>
It executes pattern matching against a value related with key
and replaces it with replace
if it matches.
Need ()
in pattern
if you want to refer values in replace
. This is Ruby's capturing feature.
/foo?bar=1 -> /foo/bar/1
rule: ignore
For example, if you want to skip a message which matches some pattern:
<rule>
key status
pattern ^500$
ignore true
</rule>
It executes pattern matching against a value related with key
and skip emitting the message if it matches.
rule: append_to_tag
<rule>
key path
pattern ^\/(users|entries)
append_to_tag true
</rule>
It executes pattern matching against a value related with key
and append mathed strings to message tag.
This also need ()
to append values to tag. See rule: replace section. For example:
apache.log { "path" : "/users/antipop" }
the messabe above will be re-emmited as the message below:
apache.log.users { "path" : "/users/antipop" }
If you set fallback
option like below:
<rule>
key path
pattern ^\/(users|entries)
append_to_tag true
fallback others
</rule>
the value of fallback
option will be appended to message tag.
apache.log { "path" : "/foo/bar" }
This time, the messabe above will be re-emmited as the message below:
apache.log.others { "path" : "/foo/bar" }
If tag
option set,
<rule>
key is_loggged_in
pattern 1
append_to_tag true
tag user
</rule>
the value designated by tag
will be appended to the original tag, that is:
test { "is_logged_in" => "1" }
will be
test.user { "is_logged_in" => "1" }
rule: last
If you set last
option to true, rewriting chain stops applying rule where the pattern matches first.
<rule>
key path
pattern ^/foo$
replace /bar
last true
</rule>
<rule>
key path
pattern ^/bar$
replace /baz
</rule>
This rules will be applied like below:
{ "path" => "/foo" }
will be replaced with
{ "path" => "/bar" }
and the chain stops here. Therefore, the second rule is never applied.
{ "path" => "/bar" }
will be replaced by the second rule as usual.
{ "path" => "/baz" }
RewriteFilter
Filter plugin to modify messages' values along with pattern matching and filter them.
Note that filter version of rewrite plugin does not have append/add tags functionality.
Thus, this filter version does not able to specify append_to_tag
, tag
, and fallback
rules.
Synopsis
<filter apache.log.**>
@type rewrite
<rule>
key path
pattern \\?.+$
replace
</rule>
<rule>
key path
pattern (/[^/]+)\\?([^=]+)=(\\d)
replace \\1/\\2/\\3
</rule>
<rule>
key status
pattern ^500$
ignore true
</rule>
</filter>
Configuration
Note: This filter version of rewrite plugin does not have remove_prefix
and add_prefix
configuration.
rule: replace
Same as OutputRewrite section's rule: replace.
rule: ignore
Same as OutputRewrite section's rule: ignore.
rule: last
Same as OutputRewrite section's rule: last.
Installation
Add this line to your application's Gemfile:
gem 'fluent-plugin-rewrite'
And then execute:
$ bundle
Or install it yourself as:
$ gem install fluent-plugin-rewrite
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Added some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request
Copyright
Copyright
Copyright (c) 2012- Kentaro Kuribayashi (@kentaro)
License
Apache License, Version 2.0