schrodingersbox/split_cat

This Rails engine provides a framework for split testing.

This framework allows you to assign anonymous users to experiments with goals and weighted hypotheses. It makes experiments easy to configure and implement. The reporting side still needs some work.

1. Add this to your Gemfile and bundle install

    gem 'split_cat'
   

2. Add this to your config/routes.rb

    mount SplitCat::Engine => '/split_cat'
   

3. Install and run migrations

    rake split_cat:install:migrations
    rake db:migrate
   

4. Generate some random data

    rake split_cat:random[100,4,4]
   

5. Restart your Rails server

6. Visit http://yourapp/split_cat in a browser for an HTML meter report

See The one line split-test, or how to A/B all the time

1. Every user is represented by a unique token.
2. Per experiment, a user may only have one hypothesis assigned.
3. Per experiment, a use may have many goals achieved.

Modify config.application.rb to inject your authorization filter into the controller:

config.after_initialize do
  SplitCat::ExperimentsController.instance_eval do
    before_filter :require_login
  end
end

Create or add to config/initializers/split_cat.rb:

SplitCat.configure do |config|

  config.experiment( :my_first_experiment, 'green vs red "buy" button' ) do |e|
    e.hypothesis( :a, 50, 'current color: green' )
    e.hypothesis( :b, 50, 'new color: red' )
    e.goal( :clicked_buy, 'user clicked the "buy" button' )
  end

  config.experiment( :my_second_experiment, 'registration flow order' ) do |e|
    # ...
  end

end

Create partials for each hypothesis. e.g. button_a.html.erb and button_b.html.erb

When rendering the partial, scope it with the experiment:

render :partial => split_cat_scoped( 'button', :my_first_experiment, token  )

This will cause the partial to use the hypothesis assigned for the user/token.

You can get a raw hypothesis symbol for logic-based experiments:

 hypothesis = split_cat_hypothesis( name, token )
 do_something if hypothesis == :a

Call split_cat_goal to record a goal achieved by a user:

split_cat_goal( :my_first_experiment, :clicked_buy, token )

Add the following before_file to automatically cookie and set @split_cat_token:

before_filter :set_split_cat_cookie

To obtain a new token:

@new_token = split_cat_token

To use an externally defined token:

split_cat_token( external_token )

You will usually want to run a series of experiments with the same structure. Using a constant makes it easy to 'bump' the experiment name and just update the views and all the hypothesis selection and goal recording just keeps working.

e.g config/initializers/split_cat.rb

HOMEPAGE_EXPERIMENT = :homepage_4

SplitCat.configure do |config|

  config.experiment( HOMEPAGE_EXPERIMENT, 'green vs red "buy" button' ) do |e|
    e.hypothesis( :a, 50, 'current color: green' )
    e.hypothesis( :b, 50, 'new color: red' )
    e.goal( :clicked_buy, 'user clicked the "buy" button' )
  end

end

e.g. views/home/index.html.erb

render :partial => split_cat_scoped( HOMEPAGE_EXPERIMENT, token, 'buy_button' )

e.g. controllers/home_controller.rb

def buy
  split_cat_goal( HOMEPAGE_EXPERIMENT, :clicked_buy, token )
end

Setting up the hypothesis in the controller lets you easily override it with a param, which is nice during development, as it allows you to directly access hypothesis views without using multiple user accounts.

before_filter :setup_hypothesis, :only => [ :index ]

def setup_hypothesis
	@hypothesis = params[ :hypothesis ]
	@hypothesis ||= split_cat_hypothesis( HOMEPAGE_EXPERIMENT, @split_cat_token )
end

• Getting Started with Engines
• Testing Rails Engines With Rspec
• How do I write a Rails 3.1 engine controller test in rspec?
• Best practice for specifying dependencies that cannot be put in gemspec?
• Clarifying the Roles of the .gemspec and Gemfile
• The Semi-Isolated Rails Engine
• FactoryGirl
• Shoulda
• The one line split-test, or how to A/B all the time
• Add Achievement Badges to Your Gem README

• Spec rake tasks

• As needed

  • Add pagination to admin
  • Add a goals controller with action to record goal and redirect
  • Add split test probability calculator
    • hypo.control - boolean flag to determine which is baseline
  • Support goals in funnel format and percentage along funnel, rather than of overall
    • goal.funnel where nil means hypo total, or reference to other goal to use as total

• Cleanup

  • Save config changes (weights, descs) through to DB
  • Archive & delete experiments
  • Fill in all the table relationships
  • Add dependent destroys

• REST API

• Document

  • How to
    • Evaluate an experiment