fluent-plugin-file-alternative

NOTE: This plugin is outdated for Fluentd v0.14 ('file' output plugin in v0.14 has all features of this plugin)

File output plugin alternative implementation.

FileAlternativeOutput slices data by time (for specified units), and store these data as plain text on local file. You can specify to:

format whole data as serialized JSON, single attribute or separated multi attributes
include time as line header, or not
include tag as line header, or not
change field separator (default: TAB)
add new line as termination, or not

And you can specify output file path as:

Standard out_file way
- configure path /path/to/dir/access
- and time_slice_format %Y%m%d
- got /path/to/dir/access.20120316.log
Standard out_file way (alternative)
- configure path /path/to/dir/access.*.log
- and time_slice_format %Y%m%d
- got /path/to/dir/access.20120316.log
Alternative style
- configure path /path/to/dir/access.%Y%m%d.log only (don't include *)
- got /path/to/dir/access.20120316.log

And, gzip compression is also supported.

Configuration

FileAlternativeOutput

Standard out_file way (hourly log, compression, time-tag-json):

<match out.**>
  type file_alternative
  path /var/log/service/access.*.log
  time_slice_format %Y%m%d_%H
  compress gzip
</match>

By this configuration, in gzip compressed file '/var/log/service/access.20120316_23.log.gz', you get:

2012-03-16T23:59:40 [TAB] out.service.xxx [TAB] {"field1":"value1","field2":"value2"}
2012-03-16T23:59:40 [TAB] out.service.xxx [TAB] {"field1":"value1","field2":"value2"}
2012-03-16T23:59:40 [TAB] out.service.xxx [TAB] {"field1":"value1","field2":"value2"}
2012-03-16T23:59:40 [TAB] out.service.xxx [TAB] {"field1":"value1","field2":"value2"}

If you don't want fluentd-time and tag in written file, and messages with single attribute (as raw full apache log with newline):

<match out.**>
  type file_alternative
  path /var/log/service/access.%Y%m%d_%H.log
  compress gzip
  output_include_time false
  output_include_tag false
  output_data_type attr:message
  add_newline false
  symlink_path /var/log/service/access_latest.log
  dir_mode 0777
</match>

Then, you will get:

192.168.0.1 - - [16/Mar/2012:23:59:40 +0900] "GET /content/x HTTP/1.1" 200 -
192.168.0.1 - - [16/Mar/2012:23:59:40 +0900] "GET /content/x HTTP/1.1" 200 -
192.168.0.1 - - [16/Mar/2012:23:59:40 +0900] "GET /content/x HTTP/1.1" 200 -

TODO

consider what to do next
patches welcome!