No commit activity in last 3 years
No release in over 3 years
included to format values into json, tsv or csv
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

>= 0

Runtime

>= 0
 Project Readme

Fluent::Mixin::PlainTextFormatter

Fluent::Mixin::PlainTextFormatter is a mix-in module, that provides '#format' instance method and its configurations to Fluentd BufferedOutput Plugin and TimeSlicedOutput Plugin, to output plain text data (to file, REST storages, KVSs ...).

This module provides features to:

  • format whole data as serialized JSON, single attribute or separated multi attributes
  • include time as line header (formatted by time_format in UTC(default) or localtime), or not
  • include tag as line header (remove_prefix available), or not
  • change field separator (TAB(default), SPACE, COMMA or SOH(\001))
  • add new line as termination, or not

Usage

To use this module in your fluentd plugin, include this module like below:

class FooOutput < BufferedOutput
  Fluent::Plugin.register_output('foo', self)
  
  config_set_default :buffer_type, 'memory'
  
  include Fluent::Mixin::PlainTextFormatter
  
  config_param :foo_config_x, :string, :default => nil

  # and other your plugin's configuration parameters...

  def configure(conf)
    super
    
    # ....
  end
  
  def start
    # ...
  end
  
  def shutdown
    # ....
  end
  
  # def format(tag, time, record)
  #   # do not define 'format'
  # end
  
  def write(chunk)
    # ....
  end
end

And you can overwrite default formatting configurations on your plugin (values below are default of mix-in):

class FooOutput < BufferedOutput
  # ...
  
  config_set_default :output_include_time, true
  config_set_default :output_include_tag, true
  config_set_default :output_data_type, 'json'
  config_set_default :field_separator, 'TAB'
  config_set_default :add_newline, true
  config_set_default :time_format, nil   # nil means ISO8601 '2012-07-13T19:29:49+09:00'
  config_set_default :remove_prefix, nil
  config_set_default :default_tag, nil
  config_set_default :null_value, 'NULL'
  
  # ...
end

Provided configurations are below:

  • output_include_time [yes/no]
  • output_include_tag [yes/no]
  • output_data_type
    • 'json': output by JSON
    • 'ltsv': output by LTSV, see: http://ltsv.org/
    • 'attr:key1,key2,key3': values of 'key1' and 'key2' and ..., with separator specified by 'field_separator'
  • field_separator [TAB/SPACE/COMMA/SOH]
  • add_newline [yes/no]
  • time_format
    • format string like '%Y-%m-%d %H:%M:%S' or you want
  • remove_prefix
    • input tag 'test.foo' with 'remove_prefix test', output tag is 'foo'.
    • 'default_tag' configuration is used when input tag is completely equal to 'remove_prefix'
  • null_value
    • output value if value is null(nil). default is 'NULL'.

AUTHOR / CONTRIBUTORS

LICENSE

  • Copyright: Copyright (c) 2012- tagomoris
  • License: Apache License, Version 2.0