fluent-plugin-insert-id

Fluentd filter plugin to insert unique ID string into the message.

The original implementation was copied from fluent-plugin-google-cloud which was created by Google to handle GCP GKE's official Stackdriver Logging reporting in sidecar container for each pod. You can check the difference between these plugins in here

# Input message
{"a": "foo", "b": "bar"}
{"a": "foo", "b": "bar"}
{"a": "foo", "b": "bar"}
{"a": "foo", "b": "bar"}
{"a": "foo", "b": "bar"}

# Output message
2019-08-25 21:20:43.539536458 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetddw"}
2019-08-25 21:20:44.625784851 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetddx"}
2019-08-25 21:20:45.579060596 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetddy"}
2019-08-25 21:20:47.019565139 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetddz"}
2019-08-25 21:20:50.035415329 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetdea"}

• Inserted ID contains 0-9 and a-z lowercase characters.
• Initial ID generated is random string like "nu8a3ptahpbetddc".
• Series ID after the initial ID are 'incremented' string which uses Ruby's String.next().
• 'incremented' string also has 'carry' feature. Please check below links for more details.
  • String.next()
  • String.next() (Japanese)
• This ordered ID makes debugging easier in most cases.

• From version 1.1.0, it is guaranteed that ID is fixed length string.
• In version 1.0.0, the ID string length is incremented when carry happens at left-most characters.

# Version 1.0.0
{"a":"foo","b":"bar","insert-id":"z99999999999999z"}
{"a":"foo","b":"bar","insert-id":"aa00000000000000z"} # Left most character carry adds new digit.

# Version 1.1.0
{"a":"foo","b":"bar","insert-id":"z99999999999999z"}
{"a":"foo","b":"bar","insert-id":"a00000000000000z"} # Left most character carry is ignored.

If the message already has the key for inserted ID, the filter doesn't touch it and existing value is protected.

2019-08-27 02:10:07.422911774 +0000 message.test: {"a":"foo","b":"bar","insert-id":"ehrbwzp772xitjsv"}
2019-08-27 02:10:08.129842499 +0000 message.test: {"a":"foo","b":"bar","insert-id":"ehrbwzp772xitjsw"}
2019-08-27 02:10:08.940316454 +0000 message.test: {"a":"foo","b":"bar","insert-id":"ehrbwzp772xitjsx"}
2019-08-27 02:11:02.498772740 +0000 message.test: {"a":"foo","b":"bar","insert-id":"existing_ID"}
2019-08-27 02:11:06.802934944 +0000 message.test: {"a":"foo","b":"bar","insert-id":"ehrbwzp772xitjsy"}

The fluent-plugin-google-cloud also provides ID insertion filter plugin as one of its function. Please note that there are some differences between these two plugins.

fluentd v0.12 is not supported.

$ gem install fluent-plugin-insert-id

Add following line to your Gemfile:

gem "fluent-plugin-insert-id"

And then execute:

$ bundle

insert_id_key (string) (optional)

The key of inserted-id. Default value is "insert-id".

<filter **>
  @type insert_id
</filter>

In this case, key of ID is changed to 'message_id'

<filter **>
  @type insert_id
  insert_id_key message_id
</filter>

• Copyright(c) 2019- Kamome Shido
• Copyright(c) 2014- Google Inc. All rights reserved.
• License
  • Apache License, Version 2.0