edhd¶ ↑

Experiment A/B framework for Ruby. Thread-safety is left as an exercise to the user. I use globals. Shoot me. Supports nested experiments.

Experiments are defined as follows (look at the specs for more details):

• Exactly one of the hypotheses blocks in an experiment will run.

• A hypothesis will be chosen with probability proportional to its weight.

• You can pass in a seed for the random number generator to ensure that, for example, a given user always sees the same hypothesis.

• An experiment block will log which hypothesis actually runs by calling $observer.hypothesized(experiment_name, hypothesis_name, seed)

• Calling observation(name, value, seed) will log to $observer.observed(name, value, seed).

• $observer = IOObserver(STDOUT) will give you a sample observer, but you’ll probably want to log to a database or something. Exercise left to the user.

• Statistical analysis is an exercise left to the user.

  # Method signatures
  # def experiment(name = "default", seed = rand, &block)
  # def hypothesis(name, weight = 1, &block)
  # def observation(name, value, seed)
  
  # Example
  experiment "hello world", current_user.login do
    hypothesis "a1", 3 do
      @a = 1
    end
  
    hypothesis "a2" do
      @a = 2
    end
  end
  
  observation "value of a", @a, current_user.login
  

Copyright © 2009 Kyle Maxwell. MIT Licensed