No commit activity in last 3 years
No release in over 3 years
Fluentd doesn't guarantee message order but you may keep message order.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

>= 0.9.6
>= 0

Runtime

 Project Readme

README

Name

fluent-plugin-sort

Description

Fluent-plugin-sort is a Fluentd plugin to sort messages in buffer. Fluentd doesn't guarantee message order but you may keep message order.

If you use large buffer, you will keep order of many messages but messages are delayed. If you use small buffer, you will keep order of some messages but messages are delayed a bit.

Install

% gem install fluent-plugin-sort

Usage

Fluentd doesn't guarantee message order. It means that messages may be out of order. Fluent-plugin-sort plugin reduces the case.

Fluent-plugin-sort keeps received messages before routing the next output plugin. Fluent-plugin-sort sorts buffered messages when buffer is flushed. It will reduce out of order messages but it's not perfect. Some messages may be out of order.

Here are sample configuration:

<match message.**>
  type sort
  add_tag_prefix sorted.
  flush_interval 60
</match>

You can reduce out of order messages by increasing flush_interval parameter in type sort configuration. If its value is large, many messages will be sorted at once. It reduces out of order messages. But large flush interval delays message routing. Tune the parameter carefully.

You can use fluent-plugin-sort for log archive system. See Log archive at project-hatohol/hatohol Wiki. It describes architectures, how to set up and how to configure.

Reference

Fluent-plugin-sort is a buffered output plugin. You can use buffer related parameters.

Included mix-ins

  • Fluent::HandleTagNameMixin: It adds the following parameters:
    • add_tag_prefix
    • add_tag_suffix
    • remove_tag_prefix
    • remove_tag_suffix

Parameters

sort_key:

Default: time

It specifies key for sort. You can sort messages by time-stamp in a fluent message or a record value.

If you want to use time-stamp in a fluent message, specify time. time is the default value.

Example:

<match>
  type sort
  sort_key time
</match>

If you want to use a record value, specify record key with attribute: prefix.

Here is an example to use id value:

<match>
  type sort
  sort_key attribute:id
</match>

Buffered records:

[
  {"id": 2},
  {"id": 4},
  {"id": 1},
  {"id": 3}
]

Sorted result:

[
  {"id": 1},
  {"id": 2},
  {"id": 3},
  {"id": 4}
]

You can use deep record value by separating by .. Using . as delimiter is inspired by JSONPath.

Here is an example to use body.time-stamp value:

<match>
  type sort
  sort_key attribute:body.time-stamp
</match>

Buffered records:

[
  {"body": {"time-stamp": 1413272109}},
  {"body": {"time-stamp": 1413272107}},
  {"body": {"time-stamp": 1413272108}},
  {"body": {"time-stamp": 1413272106}}
]

Sorted result:

[
  {"body": {"time-stamp": 1413272106}},
  {"body": {"time-stamp": 1413272107}},
  {"body": {"time-stamp": 1413272108}},
  {"body": {"time-stamp": 1413272109}}
]

If attribute value is null or attribute doesn't exist, the message is treated as the most oldest message.

Buffered records:

[
  {"body": {"time-stamp":    1413272109}},
  {"body": {"time-stamp":    1413272107}},
  {"body": {"time-stamp":    1413272108}},
  {"body": {"no-time-stamp": true}}
]

Sorted result:

[
  {"body": {"no-time-stamp": true}},
  {"body": {"time-stamp":    1413272107}},
  {"body": {"time-stamp":    1413272108}},
  {"body": {"time-stamp":    1413272109}}
]

Authors

  • Project Hatohol

License

LGPL 2.1 or later. See doc/text/lgpl-2.1.txt for details.

Place to share information of fluent-plugin-sort

Use GitHub issues to submit a question and bug report. You can use English or Japanese on it.

Source

The repository for fluent-plugin-sort is on GitHub.

Thanks

  • ...