fluent-plugin-azurefunctions

Azure Functions (HTTP Trigger) output plugin for Fluentd. The plugin aggregates semi-structured data in real-time and writes the buffered data via HTTPS request to HTTP Trigger Function.

$ gem install fluent-plugin-azurefunctions

Create a function (HTTP Trigger). First, you need to have an function app that hosts the execution of your functions in Azure if you don't already have. Once you have an function app, you can create a function. Here are instructions:

• Create your first Azure Function
• Azure Functions developer reference

A quick-start HTTP trigger function sample is included under examples/function-csharp:

examples/function-csharp
  - function.json    (Configurations for Azure Functions)
  - project.json     (Configurations for package dependencies)
  - run.csx          (Function code)

<match azurefunctions.**>
    @type azurefunctions
    @log_level info
    endpoint  AZURE_FUNCTION_ENDPOINT   # ex. https://<accountname>.azurewebsites.net/api/<functionname>
    function_key AZURE_FUNCTION_KEY     # ex. aRVQ7Lj0vzDhY0JBYF8gpxYyEBxLwhO51JSC7X5dZFbTvROs7uNg==
    key_names key1,key2,key3
    add_time_field true
    time_field_name mytime
    time_format %s
    localtime true
    add_tag_field true
    tag_field_name mytag
</match>

• endpoint (required) - Azure Functions Endpoint URL
• function_key (required) - Azure Functions API Key
• key_names (optional) - Key names in incoming records to post to Azure functions HTTP Trigger functions. Each key needs to be separated by a comma. If key_names not specified, all incoming records are posted. If incoming records contain the same key names as the ones specified in time_field_name and tag_field_name, their values are replaced by the values of time_field_name and tag_field_name
• add_time_field (optional) - Default:true. This option allows to insert a time field to record
• time_field_name (optional) - Default:time. This is required only when add_time_field is true
• localtime (optional) - Default:false. Time record is inserted with UTC (Coordinated Universal Time) by default. This option allows to use local time if you set localtime true. This is valid only when add_time_field is true
• time_format (optional) - Default:%s. Time format for a time field to be inserted. Default format is %s, that is unix epoch time. If you want it to be more human readable, set this %Y%m%d-%H:%M:%S, for example. This is valid only when add_time_field is true.
• add_tag_field (optional) - Default:false. This option allows to insert a tag field to record
• tag_field_name (optional) - Default:tag. This is required only when add_time_field is true

[note] @log_level is a fluentd built-in parameter (optional) that controls verbosity of logging: fatal|error|warn|info|debug|trace (See also Logging of Fluentd)

fluent-plugin-azurefunctions adds .rid attribute which is UUID format and any other attributes of record automatically. In addition, it adds time and tag attributes if add_time_field and add_tag_field are true respectively. Below are two types of the plugin configurations - Default and All options configuration.

fluent.conf

<source>
    @type forward
    @id forward_input
</source>

<match azurefunctions.**>
    @type azurefunctions
    endpoint https://yoichikademo.azurewebsites.net/api/HttpTriggerFunction
    function_key aRVQ7Lj0vzDhY0JBYF8gpxYyEBxLwhO51JSC7X5dZFbTvROs7uNg==(dummy)
</match>

The plugin write all records in incoming event stream out to Azure Functions:

# Generating test event using fluent-cat
echo ' { "key1":"value1", "key2":"value2", "key3":"value3"}' | fluent-cat azurefunctions.msg

# HTTP POST request body to Azure Functions
{
    "payload": '{"key1":"value1", "key2":"value2", "key3":"value3", "time":"1479741633"}'
}

fluent.conf

<source>
    @type forward
    @id forward_input
</source>

<match azurefunctions.**>
    @type azurefunctions
    endpoint https://yoichikademo.azurewebsites.net/api/HttpTriggerFunction
    function_key aRVQ7Lj0vzDhY0JBYF8gpxYyEBxLwhO51JSC7X5dZFbTvROs7uNg==(dummy)
    key_names key1,key2
    add_time_field true
    time_field_name mytime
    time_format %s
    localtime true
    add_tag_field true
    tag_field_name mytag
</match>

The plugin write only records that are specified by key_names in incoming event stream out to Azure Functions:

# Generating test event using fluent-cat
echo ' { "key1":"value1", "key2":"value2", "key3":"value3"}' | fluent-cat azurefunctions.msg

# HTTP POST request body to Azure Functions
{
    "payload": '{"key1":"value1", "key2":"value2", "time":"1479741633", "tag":"azurefunctions.msg"}'
}

$ git clone https://github.com/yokawasa/fluent-plugin-azurefunctions.git
$ cd fluent-plugin-azurefunctions

# edit CONFIG params of test/plugin/test_azurefunctions.rb
$ vi test/plugin/test_azurefunctions.rb

# run test
$ rake test

$ rake build
$ rake install:local

# running fluentd with your fluent.conf
$ fluentd -c fluent.conf -vv &

# generate test event
$ echo ' { "key1":"value1", "key2":"value2", "key3":"value3"}' | fluent-cat azurefunctions.msg

• Changelog

• https://yokawasa.github.io/fluent-plugin-azurefunctions
• https://rubygems.org/gems/fluent-plugin-azurefunctions
• http://unofficialism.info/posts/fluent-plugin-azurefunctions/

Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/fluent-plugin-azurefunctions.