Cassandra plugin for Fluentd
Cassandra output plugin for Fluentd.
Implemented using the Datastax Ruby Driver for Apache Cassandra gem and targets CQL3 and Cassandra 1.2 - 3.x
Warning
This project is in an alpha state, so configuration params could be changed without changing of major version.
Be careful before updating.
Installation
via RubyGems
fluent-gem install fluent-plugin-cassandra-driver
td-agent-gem install fluent-plugin-cassandra-driver
Quick Start
Cassandra Configuration
# Create keyspace (via CQL)
CREATE KEYSPACE metrics WITH strategy_class='org.apache.cassandra.locator.SimpleStrategy' AND strategy_options:replication_factor=1;
# Create table (column family)
CREATE TABLE logs (id varchar, timestamp timestamp, json text, PRIMARY KEY (id, timestamp)) WITH CLUSTERING ORDER BY (timestamp DESC);
Fluentd.conf Configuration
<match cassandra.**>
type cassandra_driver # fluent output plugin file name (sans fluent_plugin_ prefix)
hosts 127.0.0.1 # comma delimited string of hosts
keyspace metrics # cassandra keyspace
column_family logs # cassandra column family
ttl 60 # cassandra ttl (optional, default is 0)
schema # cassandra column family schema (see example below)
pop_data_keys # pop values from the fluentd hash when storing it as json (optional, default is true)
json_column json # column where store all remaining data from fluentd (optional)
</match>
Schema example
# hash of hashes :column_damily_key => {:fluentd_record_key => :type_from_list}
# or :column_damily_key => :type_from_list
# then :fluentd_record_key will be the same as :column_damily_key
'{:id => {:ident => nil}, :timestamp => {:timestamp => :time}}'
Available mappings:
- :integer
- :string
- :timeuuid
- :time
All nil types will be recognized as string.
Tests
TODO