fluent-plugin-dedup

A fluentd plugin to suppress emission of subsequent logs identical to the first one.

It's useful when the output of a command executed by in_exec only returns the "latest" state of something and you want to send logs only when there is a change.

<source>
  type exec
  command latest_state_of_something.rb
  format json
  keys unique_id,foo,bar
  tag some.thing
  run_interval 1s
</source>

<match some.thing>
  type dedup
  key  unique_id # required
  file /tmp/dedup_state.json # optional. If set, saves the state to the file.
  cache_per_tag 10 # optional. If set, recent logs up to this number is cached.
  cache_ttl 600 # optional. If set, cache entries is expired in this TTL.
</match>

<match dedup.some.thing>
  type stdout
</match>

All logs that are processed by this plugin will have tag prefix dedup.

If the optional file parameter is set, it dumps the state during shutdown and loads on start, so that it can still dedup after reload.

If the optional cache_per_tag is set, it caches N recently appeared logs (only caches unique_id in this example) and compared to new logs.

If the optional cache_ttl is set, it evicts cache entries in a specific amount of time.

bundle install
bundle exec rake test

gem install fluent-plugin-dedup

1. Fork it ( https://github.com/[my-github-username]/fluent-plugin-dedup/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request