No commit activity in last 3 years
No release in over 3 years
A Resque plugin. Allows you to use Redis to queue up and then Resque to process transactions on transaction-heavy tables that need to be replicated on other tables optimized for reporting.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Runtime

< 1.6, >= 1.4.6
>= 3.0.0
>= 1.9.10
 Project Readme

Resque Data Warehouse

A Resque plugin. Requires Resque 1.9.10 or higher.

resque-data-warehouse allows you to use Redis to queue up and then Resque to process transactions on transaction-heavy tables that need to be replicated on other tables optimized for reporting.

Transactions for a given object (classname + ID) are queued up behind a Redis key, and then processed using Resque jobs. If load is low, each transaction will be processed almost immediately after it occurs; at higher loads, multiple transactions will queue up before the Resque job gets to them, and then only the cumulative result of the previous transactions will be applied to the data warehousing table, thus minimizing database load and dynamically adjusting the delay in the copy to match the current load.

This only works with Rails >=3.0; it has been tested with Rails 3.0.5.

Usage / Examples

Suppose you have a database class Transactional, that gets a lot of traffic, and you want to have a counterpart to this class, TransactionalFact, in a data warehouse. All you need to do are the following:

  • Create an ActiveRecord model named TransactionalFact, with all the instance variables you want in it.
  • Add a method to TransactionalFact named execute_transaction, which assumes that all the fields in the fact that match the original Transactional are already set, saves them, and then performs any additional logic (denormalization, etc.) to update any remaining fields.
  • Require 'resque-data-warehouse' in Transactional, and add a line "warehoused".

Very simple examples of both classes:

class Transactional < ActiveRecord::Base
  require 'resque-data-warehouse'
  warehoused
end

class TransactionalFact < ActiveRecord::Base
  def execute_transaction
    self.save
    # Any additional logic here
  end
end

Customize & Extend

No customizations are available right now, but some obvious ones (such as how to derive the name of the warehoused class) are likely to occur soon.

Install

As a gem

$ gem install resque-data-warehouse

In a Rails app, as a plugin

$ ./script/plugin install git://github.com/Mechaferret/resque-data-warehouse.git

Acknowledgements

Thanks to resque and Redis for making this work possible.