Repository is archived
No commit activity in last 3 years
No release in over 3 years
A fluentd redis input plugin supporting batch operations
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

>= 0

Runtime

< 2, >= 0.14.0
< 3.4.0, >= 3.3.0
 Project Readme

Abandoned Project

NOTE: This project is no longer being maintained.

Fluent::Plugin::RedisListPoller

This gem provides a few input plugins for Fluentd which are designed to more efficiently poll large volumes of messages than other existing redis input plugins.

  • redis_list_poller: Input plugin designed to fetch large volumes of messages
  • redis_list_monitor: Input plugin designed to fetch queue size metrics

In additional to the standard stuff, the redis_list_poller input plugin also looks for a lock key in Redis to see whether it has been administratively disabled. This lock key follows the naming convention of redis:KEY_NAME:lock.

Requirements

  • Fluentd v0.14+

Installation

gem install fluent-plugin-redis_list_poller

Configuration Options

<source>
  @type       redis_list_poller
  host        127.0.0.1
  port        6379
  path        nil
  password    nil
  db          0
  timeout     5.0
  driver      ruby

  key         redis_list_item
  command     lpop
  batch_size  100

  tag         redis.data

  poll_interval      0.01
  sleep_interval     5
  retry_interval    5

  <parse>
    @type json
  </parse>
</source>

Benchmarks

These very simple benchmarks which give a rough ideas of the sorts of message rates one can expect from this input plugin.

They were run within a Vagrant box, with a local Redis, running on my Macbook Pro 15'. Fluentd is running on a single CPU core and is using the examples/standalone.conf.

poll_interval batch_size messages/second
1 1 1
0.5 1 2
0.2 1 5
0.01 1 100
0.001 1 457
poll_interval batch_size messages/second
1 10 10
0.5 10 20
0.2 10 50
0.01 10 1000
0.001 10 1793
poll_interval batch_size messages/second
1 100 100
0.5 100 200
0.2 100 500
0.01 100 7823
0.001 100 7600