Fluent::Plugin::Anomalydetect, a plugin for Fluentd

To detect anomaly for log stream, use this plugin. Then you can find changes in logs casually.

Installation

Add this line to your application's Gemfile:

gem 'fluent-plugin-anomalydetect'

And then execute:

$ bundle

Or install it yourself as:

$ gem install fluent-plugin-anomalydetect

Usage

<source>
  type file
  ...
  tag access.log
</source>

<match access.**>
  type anomalydetect
  tag anomaly.access
  tick 86400
</match>

<match anomaly.access>
  type file
  ...
</match>

Then the plugin output anomaly log counts in each day.

This plugin watches a value of input record number in the interval set with tick.

If you want to watch a value for a target field in data, write below:

<match access.**>
  type anomalydetect
  tag anomaly.access
  tick 86400
  target fieldname
</match>

more configuration

<match access.**>
  type anomalydetect
  tag anomaly.access
  tick 86400
  target fieldname
  outlier_term 7
  outlier_discount 0.5
  smooth_term 7
  score_term 28
  score_discount 0.01
</match>

If you want to know detail of these parameters, see "Theory".

<match access.**>
  type anomalydetect
  ...
  store_file /path/to/anomalydetect.dat
</match>

If "store_file" option was specified, a historical stat will be stored to the file at shutdown, and it will be restored on started.

<match access.**>
  type anomalydetect
  ...
  threshold 3
</match>

If "threshold" option was specified, plugin only ouput when the anomalyscore is more than threshold.

<match access.**>
  type anomalydetect
  ...
  trend up
</match>

If "trend" option was specified, plugin only ouput when the input data tends to up (or down).

Parameters

outlier_term
outlier_discount
smooth_term
score_term
score_discount
tick

The time interval to watch in seconds.
tag

The output tag name. Required for aggregate all. Default is anomaly.
add_tag_prefix

Add tag prefix for output message. Required for aggregate tag.
remove_tag_prefix

Remove tag prefix for output message.
aggragate

Process data for each tag or all. The default is all.
target

Watch a value of a target field in data. If not specified, the number of records is watched (default). The output would become like:
```
  {"outlier":1.783,"score":4.092,"target":10}
```
threshold

Emit message only if the score is greater than the threshold. Default is -1.0.
trend

Emit message only if the input data trend is up (or down). Default is nil.
store_file

Store the learning results into a file, and reload it on restarting.
targets

Watch target fields in data. Specify by comma separated value like x,y. The output messsages would be like:
```
  {"x_outlier":1.783,"x_score":4.092,"x":10,"y_outlier":2.310,"y_score":3.982,"y":3}
```
thresholds

Threahold values for each target. Specify by comma separated value like 1.0,2.0. Use with targets option.
outlier_suffix

Change the suffix of emitted messages of targets option. Default is _outlier.
score_suffix

Change the suffix of emitted messages of targets option. Default is _score.
target_suffix

Change the suffix of emitted messages of targets option. Default is `` (empty).
suppress_tick

Suppress to emit output messsages during specified seconds after starting up.