fluent-plugin-out-kaboom
A Fluentd plugin for exploding JSON array elements
Configuration Options
| Argument | Description | Required? | Default |
|---|---|---|---|
| key | The key of the array to explode | Yes | N/A |
| tag | The tag to use for emitted messages | No | Defers to add_tag_prefix or remove_tag_prefix |
| remove_tag_prefix | The prefix to remove from the tags of emitted messages | No | N/A |
| add_tag_prefix | The prefix to add to the tags of emitted messages | No | N/A |
If you do not specify tag, you must specify remove_tag_prefix, add_tag_prefix, or both. remove_tag_prefix will be applied before add_tag_prefix.
Example Usage
Consider a Fluentd message with the tag users and the following contents:
{"user": {"first_name": "John", "last_name": "Smith", "favorite_movies": ["John Wick", "Robocop", "Blade Runner"]}}
If you need to run analytics on this data via a database like Redshift, which does not support Arrays, favorite_movies needs to be exploded. We can use a Fluentd configuration like this:
<match users>
@type kaboom
key user.favorite_movies
add_tag_prefix exploded.
</match>
This will result in three new messages being emitted:
{"user": {"first_name": "John", "last_name": "Smith", "favorite_movies": "John Wick"}}{"user": {"first_name": "John", "last_name": "Smith", "favorite_movies": "Robocop"}}{"user": {"first_name": "John", "last_name": "Smith", "favorite_movies": "Blade Runner"}}
Each new message will be tagged exploded.users adhering to the add_tag_prefix configuration value. This tag can then be matched on later enabling processing of the messages individually. To complete the use case mentioned earlier, they can be put into Redshift so that the userbase's favorite movies can be analyzed.