Transparently maintains an attribute of an ActiveRecord-backed model in a separate key-value data store such as Amazon S3, memcached, Tokyo Cabinet, or the filesystem.

HashPipe makes it easy to use an external key-value based data storage system alongside a traditional ActiveRecord- backed model. In this sense, it acts as a pipe from the ActiveRecord model to a hash - a hash pipe!

It functions by hooking into the native callbacks provided by ActiveRecord models. HashPipe keeps track of when the hashed attribute has been written to, and flushes it out to the external data source on saves of the ActiveRecord model. Hashed attributes are lazy-loaded, so they aren’t pulled from the external data source until they’re requested. Finally, when the ActiveRecord model is deleted, the hashed attributes for that model are also deleted automatically.

HashPipe supports transparent gzip compression/decompression of attribute data and serialization/deserialization using Marshal in case you wish to store Ruby objects in the attribute.

The backend of HashPipe is implemented using Moneta, which is a universal library for key-value storage systems.

This means that it supports all of the backend storage types supported by this common library, and that new backends are added regularly.

There are certain cases where it makes more sense to store data objects in a separate storage system from the RDBMS:

• Certain parts of your data set require frequent reads and writes, and need to be fast without the overhead introduced by the ACID properties of traditional relational database systems

• You store large objects, and don’t want to worry about increasing the size of your RDBMS when the size grows out of the space you’ve allocated

Getting started with HashPipe is easy.

• Install the gem ‘hashpipe’ available from github as jsl-hashpipe. If you’re using a recent version of Rails you may want to add a line like the following to your environment.rb:

  config.gem "jsl-hashpipe", :lib => 'hashpipe', :source => 'http://gems.github.com'
  

• If you wanted to serialize an attribute ‘elephant’ in model Foo, you would call hattr :elephant in your model.

  For example, class ‘Foo’ may look like:

  class Foo < ActiveRecord::Base
    hattr :elephant
  end
  

• Configure hashpipe by creating a file hashpipe.yml in your projects’ config directory according to the instructions in the ‘Configuration’ section of this document

• After following the above steps, saving to the model and retrieving from it saves to the backend store defined in your options. Likewise, deleting the model triggers a callback to delete the element in the associated store.

HashPipe supports the backends supported by Moneta. Configure the backend that you want to use in your YAML configuration file ‘hashpipe.yml’:

test:
  moneta_klass: Moneta::Memory

development:
  moneta_klass: Moneta::Memcache
  moneta_options: 
    server: localhost:1978

You can define a default backend globally, and override it in individual models as needed (e.g., store one attribute in the filesystem and another in S3).

Justin S. Leitgeb, <justin@phq.org>