fluent-plugin-json
Overview
Fluentd filter plugin for JSON with JSON pointer support (RFC-6901).
Installation
RubyGems
gem install fluent-plugin-json
Bundler
Add the following line to your Gemfile:
gem 'fluent-plugin-json'
And then execute:
bundle
Configuration
<check>
section (required) (multiple)
-
pointer
(string) (required): The JSON pointer to an element. -
pattern
(regexp) (required): The regular expression to match the element.
The configuration may consist of one or more checks. Each check contains a
pointer
to a JSON element and its corresponding pattern
(regex) to test it.
The checks are evaluated sequentially. The failure of a single check results in the rejection of the event. A rejected event is not routed for further processing.
NOTE: The JSON element pointed to by the pointer
is always converted to a
string for testing with the pattern
(regex).
For the detailed syntax of:
Example
Here is a sample configuration with
forward
input plugin, json
filter plugin with multiple checks and the routing to
stdout
output plugin:
<source>
@type forward
@id forward_input
</source>
<filter debug.test>
@type json
@id json_filter
<check>
pointer /log/user # point to { "log": { "user": "test", ... } }
pattern /test/i # check it against the value of username `test` (ignore case)
</check>
<check>
pointer /log/codes/0 # point to { "log": { "codes": [123, ...] } }
pattern /123/ # check it against the value at 0th index of the codes array i.e. `123`
</check>
<check>
pointer /log/level # point to { "log": { "level": "info", ... } }
pattern /.*/ # check it against all the log levels
</check>
</filter>
<match debug.test>
@type stdout
</match>
For a JSON message:
{ "log": {"user": "test", "codes": [123, 456], "level": "info"} }
Sent using fluent-cat
with tag debug.test
:
echo '{ "log": {"user": "test", "codes": [123, 456], "level": "info"} }' | fluent-cat "debug.test"
After passing all the checks, the routed event to stdout
would be:
2020-07-23 22:36:06.093187459 +0500 debug.test: {"log":{"user":"test","codes":[123,456],"level":"info"}}
By default, the checks are logged in debug
mode only:
2020-07-23 22:47:33 +0500 [debug]: #0 [json_filter] check: pass [/log/user -> 'test'] (/test/)
2020-07-23 22:47:33 +0500 [debug]: #0 [json_filter] check: pass [/log/codes/0 -> '123'] (/123/)
2020-07-23 22:47:33 +0500 [debug]: #0 [json_filter] check: pass [/log/level -> 'info'] (/.*/)
2020-07-23 22:47:33.577900915 +0500 debug.test: {"log":{"user":"test","codes":[123,456],"level":"info"}}
Contribute
- Fork the project.
- Check out the latest
main
branch. - Create a feature or bugfix branch from
main
. - Commit and push your changes.
- Make sure to add and run tests locally:
bundle exec rake test
. - Run
rubocop
locally and fix all the lint warnings. - Submit the PR.