fluent-plugin-grepcounter

Fluentd plugin to count the number of matched messages, and emit if exeeds the threshold.

Assume inputs from another plugin are as belows:

syslog.host1: {"message":"20.4.01/13T07:02:11.124202 INFO GET /ping" }
syslog.host1: {"message":"20.4.01/13T07:02:13.232645 WARN POST /auth" }
syslog.host1: {"message":"20.4.01/13T07:02:21.542145 WARN GET /favicon.ico" }
syslog.host1: {"message":"20.4.01/13T07:02:43.632145 WARN POST /login" }

An example of grepcounter configuration:

<match syslog.**>
  type grepcounter
  count_interval 60
  input_key message
  regexp WARN
  exclude favicon.ico
  threshold 1
  add_tag_prefix warn.count
</match>

Then, output bocomes as belows (indented):

warn.count.syslog.host1: {
  "count":2,
  "message":["20.4.01/13T07:02:13.232645 WARN POST /auth","20.4.01/13T07:02:43.632145 WARN POST /login"],
  "input_tag":"syslog.host1",
  "input_tag_last":"host1",
}

As default, the grepcounter plugin outputs matched message as an array as shown above. You may want to output message as a string, then use delimiter option like:

<match syslog.**>
  type grepcounter
  count_interval 60
  input_key message
  regexp WARN
  exclude favicon.ico
  threshold 1
  add_tag_prefix warn.count
  delimiter \n
</match>

Then, output bocomes as belows (indented). You can see the message field is joined with \n.

warn.count.syslog.host1: {
  "count":2,
  "message":"20.4.01/13T07:02:13.232645 WARN POST /auth\n20.4.01/13T07:02:43.632145 WARN POST /login",
  "input_tag":"syslog.host1",
  "input_tag_last":"host1",
}

• count_interval

  The interval time to count in seconds. Default is 60.

• input_key field_key

  The target field key to grep out. Use with regexp or exclude.

• regexp regexp

  The filtering regular expression

• exclude regexp

  The excluding regular expression like grep -v

• regexp[1-20] field_key regexp (experimental)

  The target field key and the filtering regular expression to grep out. No message is outputted in this case.

• exclude[1-20] field_key regexp (experimental)

  The target field key and the excluding regular expression like grep -v. No message is outputted in this case.

• threshold

  The threshold number to emit. Emit if count value >= specified value.

• greater_equal

  This is same with threshold option. Emit if count value is greater than or equal to (>=) specified value.

• greater_than

  Emit if count value is greater than (>) specified value.

• less_than

  Emit if count value is less than (<) specified value.

• less_equal

  Emit if count value is less than or equal to (<=) specified value.

• tag

  The output tag. Required for aggregate all.

• add_tag_prefix

  Add tag prefix for output message

• remove_tag_prefix

  Remove tag prefix for output message

• add_tag_suffix

  Add tag suffix for output message

• remove_tag_suffix

  Remove tag suffix for output message

• remove_tag_slice min..max

  Remove tag parts by slice function. FYI: This option behaves like tag.split('.').slice(min..max).

  For example,

    remove_tag_slice 0..-2
  

  changes an input tag foo.bar.host1 to foo.bar.

• aggregate

  Aggregation unit. One of all, in_tag, out_tag can be specified. Default is all.

  • all counts summation for all input messages and emit one message in each interval.
  • in_tag counts summation for each input tag seperately.
  • out_tag counts summation for each tag modified by add_tag_prefix, remove_tag_prefix, or remove_tag_slice.

• delimiter

  Output matched messages after joined with the specified delimiter.

• replace_invalid_sequence

  Replace invalid byte sequence in UTF-8 with '?' character if true

• store_file

  Store internal count data into a file of the given path on shutdown, and load on statring.

See CHANGELOG.md for details.

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Copyright (c) 2013 Naotoshi SEO. See LICENSE for details.