No commit activity in last 3 years
No release in over 3 years
Fluent output plugin for Cassandra via CQL version 3.0.0
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

>= 1.0.0
>= 1.8.4
>= 3.12
>= 0

Runtime

>= 0.10.28
 Project Readme

Cassandra plugin for Fluentd

Cassandra output plugin for Fluentd.

Implemented using the cassandra-cql gem and targets CQL 3.0.0 and Cassandra 1.1.x

Raison d'ĂȘtre

Currently, there's another Fluentd Cassandra plugin

It's implemented via the Twitter Cassandra gem, which:

a) doesn't provide all of the niceties of CQL, i.e., create/alter/delete keyspaces/columnfamilies
b) doesn't allow a desktop client to make a call to a Cassandra instance hosted on EC2
   (the gem resolves a cassandra node's IP address to its private EC2
    IP address (ex: 10.x.x.x), which isn't accessible outside EC2)

Installation

via RubyGems

fluent-gem install fluent-plugin-cassandra-cql

Quick Start

Cassandra Configuration

# create keyspace (via CQL)
  CREATE KEYSPACE \"FluentdLoggers\" WITH strategy_class='org.apache.cassandra.locator.SimpleStrategy' AND strategy_options:replication_factor=1;

# create table (column family)
  CREATE TABLE events (id varchar, ts bigint, payload text, PRIMARY KEY (id, ts)) WITH CLUSTERING ORDER BY (ts DESC);

# NOTE: schema definition should match that specified in the Fluentd.conf configuration file (see below)

Fluentd.conf Configuration

<match cassandra_cql.**>
  type cassandra_cql         # fluent output plugin file name (sans fluent_plugin_ prefix)
  host 127.0.0.1             # cassandra hostname.
  port 9160                  # cassandra thrft port.
  keyspace FluentdLoggers    # cassandra keyspace
  columnfamily spec_events   # cassandra column family
  ttl 60                     # cassandra ttl *optional => default is 0*
  schema                     # cassandra column family schema *hash where keys => column names and values => data types*
  data_keys                  # comma delimited string of the fluentd hash's keys
  pop_data_keys              # keep or pop key/values from the fluentd hash when storing it as json
</match>

Tests

rake rspec

NOTE: requires that cassandra be installed on the machine running the
      test as well as a keyspace named "FluentdLoggers" and a column family
      named "spec_events"

TODOs

1) make host and port configurable for tests
2) add rake task to generate keyspace and columnfamily
3) support multiple ip addresses in the connection string for Cassandra multi-node environments
4) make the cql version configurable