Replication Aware Master Slave Adapter

Improved version of the master_slave_adapter plugin, packaged as a gem.

1. automatic selection of master or slave connection: with_consistency
2. manual selection of master or slave connection: with_master, with_slave
3. handles master unavailable scenarios gracefully
4. transaction callbacks: on_commit, on_rollback
5. also:

• support for multiple slaves
• (partial) support for database_cleaner

The adapter will run all reads against a slave database, unless a) the read is inside an open transaction or b) the adapter determines that the slave lags behind the master relative to the last write. For this to work, an initial initial consistency requirement, a Clock, must be passed to the adapter. Based on this clock value, the adapter determines if a (randomly chosen) slave meets this requirement. If not, all statements are executed against master, otherwise, the slave connection is used until either a transaction is opened or a write occurs. After a successful write or transaction, the adapter determines a new consistency requirement, which is returned and can be used for subsequent operations. Note that after a write or transaction, the adapter keeps using the master connection.

As an example, a Rails application could run the following function as an around_filter:

def with_consistency_filter
  if logged_in?
    clock = cached_clock_for(current_user)

    new_clock = ActiveRecord::Base.with_consistency(clock) do
      # inside the controller, ActiveRecord models can be used just as normal.
      # The adapter will take care of choosing the right connection.
      yield
    end

    [ new_clock, clock ].compact.max.tap do |c|
      cache_clock_for(current_user, c)
    end if new_clock != clock
  else
    # anonymous users will have to wait until the slaves have caught up
    with_slave { yield }
  end
end

Note that we use the last seen consistency for a given user as reference point. This will give the user a recent view of the data, possibly reading from master, and if no write occurs inside the with_consistency block, we have a reasonable value to cache and reuse on subsequent requests. If no cached clock is available, this indicates that no particular consistency is required. Any slave connection will do. Since with_consistency blocks can be nested, the controller code could later decide to require a more recent view on the data.

See also this blog post for a more detailed explanation.

The original functionality of the adapter has been preserved:

ActiveRecord::Base.with_master do
  # everything inside here will go to master
end

ActiveRecord::Base.with_slave do
  # everything inside here will go to one of the slaves
  # opening a transaction or writing will switch to master
  # for the rest of the block
end

with_master, with_slave as well as with_consistency can be nested deliberately.

Due to scenarios when the master is possibly down (e.g., maintenance), we try to delegate as much as possible to the active slaves. In order to accomplish this we have added the following functionalities.

• We ignore errors while connecting to the master server.
• ActiveRecord::MasterUnavailable exceptions are raised in cases when we need to use a master connection, but the server is unavailable. This exception is propagated to the application.
• We have introduced the circuit breaker pattern in the master reconnect logic to prevent excessive reconnection attempts. We block any queries which require a master connection for a given timeout (by default, 30 seconds). After the timeout has expired, any attempt of using the master connection will trigger a reconnection.
• The master slave adapter is still usable for any queries that require only slave connections.

This feature was originally developed at SoundCloud for the standard MysqlAdapter. It allows arbitrary blocks of code to be deferred for execution until the next transaction completes (or rolls back).

irb> ActiveRecord::Base.on_commit { puts "COMMITTED!" }
irb> ActiveRecord::Base.on_rollback { puts "ROLLED BACK!" }
irb> ActiveRecord::Base.connection.transaction do
irb*   # ...
irb> end
COMMITTED!
=> nil
irb> ActiveRecord::Base.connection.transaction do
irb*   # ...
irb*   raise "failed operation"
irb> end
ROLLED BACK!
# stack trace omitted
=> nil

Note that a transaction callback will be fired only once, so you might want to do:

class MyModel
  after_save do
    connection.on_commit do
      # ...
    end
  end
end

The adapter keeps a list of slave connections (see Configuration) and chooses randomly between them. The selection is made at the beginning of a with_slave or with_consistency block and doesn't change until the block returns. Hence, a nested with_slave or with_consistency might run against a different slave.

At SoundCloud, we're using database_cleaner's 'truncation strategy' to wipe the database between cucumber 'feature's. As our cucumber suite proved valuable while testing the with_consistency feature, we had to support truncate_table as an ActiveRecord::Base.connection instance method. We might add other strategies if there's enough interest.

MasterSlaveAdapter requires ActiveRecord with a version >= 2.3, is compatible with at least Ruby 1.8.7, 1.9.2, 1.9.3 and comes with built-in support for mysql and mysql2 libraries.

You can check the versions it's tested against at Travis CI.

Using plain rubygems:

$ gem install master_slave_adapter

Using bundler, just include it in your Gemfile:

gem 'master_slave_adapter'

Example configuration for the development environment in database.yml:

development:
  adapter: master_slave          # use master_slave adapter
  connection_adapter: mysql      # actual adapter to use (only mysql is supported atm)
  disable_connection_test: false # when an instance is checked out from the connection pool,
                                 # we check if the connections are still alive, reconnecting if necessary

  # these values are picked up as defaults in the 'master' and 'slaves' sections:
  database: aweapp_development
  username: aweappuser
  password: s3cr3t

  master:
    host: masterhost
    username: readwrite_user     # override default value

  slaves:
    - host: slave01
    - host: slave02

You can execute all tests against your current ruby version via:

rake spec

In case you have rvm installed, you can test against 1.8.7, 1.9.2 and 1.9.3 as well as ActiveRecord 2 and 3 via:

bash spec/all.sh

• Maurício Lenhares - original master_slave_adapter plugin
• Torsten Curdt - with_consistency, maintainership & open source licenses
• Sean Treadway - chief everything & transaction callbacks
• Kim Altintop - strong lax monoidal endofunctors
• Omid Aladini - chief operator & everything else
• Tiago Loureiro - review expert & master unavailable handling
• Tobias Schmidt - typo master & activerecord ranter