Scribe input plugin for Fluentd¶ ↑

This is a plugin for fluentd data collector. This plugin adds the Scribe compatible interface to fluentd.

Scribe is a server for aggregating log data streamed in real time from a large number of servers, developed at Facebook.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

The Scribe plugin for fluentd, which enables fluentd to talk the Scribe protocol. Scribe protocol is defined as follows, in Thrift-IDL format:

enum ResultCode
{
  OK,
  TRY_LATER
}

struct LogEntry
{
  1:  string category,
  2:  string message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list<LogEntry> messages);
}

The category field is used as fluentd ‘tag’.

fluent-plugin-scribe contains both input and output.

Please add the following configurations to fluent.conf. This allows your Scribe client to import logs through port 1463.

# Scribe input
<source>
  type scribe
  port 1463
</source>

These options are supported.

• port: port number (default: 1463)

• bind: bind address (default: 0.0.0.0)

• server_type: server architecture either in ‘simple’, ‘threaded’, ‘thread_pool’, ‘nonblocking’ (default: nonblocking)

• is_framed: use framed protocol or not (default: true)

• add_prefix: prefix string, added to the tag (default: nil)

• msg_format: format of the messages either in ‘text’, ‘json’, ‘url_param’ (default: text)

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Scribe server. Note that fluentd conveys semi-structured data while Scribe conveys unstructured data, thus, ‘field_ref’ parameter is prepared to specify which field will be transferred.

# Scribe output
<match *>
  type scribe
  host scribed-host.local
  port 1463
  field_ref message
</match>

These options are supported.

• host: host name or address (default: localhost)

• port: port number (default: 1463)

• field_ref: field name which sent as scribe log message (default: message)

• timeout: thrift protocol timeout (default: 30)

• format_to_json: if true/yes, format entire record as json, and send as message (default: false)

• remove_prefix: prefix string, removed from the tag (default: nil)

To run fluentd with this plugin on chaging,

$ bundle # (or 'bundle update')
$ bundle exec fluentd -v -v -v -c example.conf

Then please execute the sample client.

$ bundle exec bin/fluent-scribe-remote

• Satoshi Tagomori

• Sadayuki Furuhashi

• Ken Robertson

Copyright

Copyright © 2011 Treasure Data, Inc.

License

Apache License, Version 2.0