fluent-plugin-groupcounter, a plugin for Fluentd

Fluentd plugin to count like SELECT COUNT(*) GROUP BY.

Assume inputs are coming as followings:

apache.access: {"code":"200", "method":"GET", "path":"/index.html", "reqtime":"1.001" }
apache.access: {"code":"202", "method":"GET", "path":"/foo.html",   "reqtime":"2.002" }
apache.access: {"code":"200", "method":"GET", "path":"/index.html", "reqtime":"3.003" }

Think of quering SELECT COUNT(\*) GROUP BY code,method,path. Configuration becomes as below:

<match apache.access>
  type groupcounter
  aggregate tag
  output_per_tag true
  add_tag_prefix groupcounter
  group_by_keys code,method,path
</match>

Output becomes like

groupcounter.apache.access: {"200_GET_/index.html_count":2, "202_GET_/foo.html_count":1}

• group_by_keys (semi-required)

  Specify keys in the event record for grouping. group_by_keys or group_by_expression is required.

• delimiter

  Specify the delimiter to join group_by_keys. Default is '_'.

• pattern[1-20]

  Use patternX option to apply grouping more roughly. For example, adding a configuration for the above example as below

     pattern1 2xx ^2\d\d
  

  gives you an ouput like

     groupcounter.apache.access: {"2xx_GET_/index.html_count":3}
  

• group_by_expression (semi-required)

  Use an expression to group the event record. group_by_keys or group_by_expression is required.

  For examples, for the exampled input above, the configuration as below

    group_by_expression ${method}${path}/${code}
  

  gives you an output like

    groupcounter.apache.access: {"GET/index.html/200_count":1, "GET/foo.html/400_count":1}
  

  SECRET TRICK: You can write a ruby code in the ${} placeholder like

    group_by_expression ${method}${path.split(".")[0]}/${code[0]}xx
  

  This gives an output like

    groupcounter.apache.access: {"GET/index/2xx_count":1, "GET/foo/4xx_count":1}
  

• tag

  The output tag. Default is groupcount.

• add_tag_prefix

  The prefix string which will be added to the input tag. output_per_tag yes must be specified together.

• remove_tag_prefix

  The prefix string which will be removed from the input tag.

• count_interval

  The interval time to count in seconds. Default is 60.

• unit

  The interval time to monitor specified an unit (either of minute, hour, or day). Use either of count_interval or unit.

• store_file

  Store internal data into a file of the given path on shutdown, and load on starting.

• max_key

  Specify key name in the event record to do SELECT COUNT(\*),MAX(key_name) GROUP BY.

  For examples, for the exampled input above, adding the configuration as below

    max_key reqtime
  

  gives you an output like

    groupcounter.apache.access: {"200_GET_/index.html_reqtime_max":3.003, "202_GET_/foo.html_reqtime_max":2.002}
  

• min_key

  Specify key name in the event record to do SELECT COUNT(\*),MIN(key_name) GROUP BY.

• avg_key

  Specify key name in the event record to do SELECT COUNT(\*),AVG(key_name) GROUP BY.

• count_suffix

  Default is _count

• max_suffix

  Default is _max. Should be used with max_key option.

• min_suffix

  Default is _min. Should be used with min_key option.

• avg_suffix

  Default is _avg. Should be used with avg_key option.

See CHANGELOG.md for details.

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

• Copyright
  • Copyright (c) 2012- Ryosuke IWANAGA (riywo)
  • Copyright (c) 2013- Naotoshi SEO (sonots)
• License
  • Apache License, Version 2.0