GOV.UK A/B Testing

Gem to help with A/B testing on the GOV.UK platform.

Installation

Add this line to your application's Gemfile:

gem 'govuk_ab_testing', '~> VERSION'

And then execute:

$ bundle

Pre-requisites

Before starting this, you'll need to:

Read the documentation for an overview on how a/b testing works on GOV.UK.
The cookie and header name in govuk-cdn-config must match the test name parameter that you pass to the Gem. The cookie name is case-sensitive.

Usage

Outline

To enable testing in the app, your Rails app needs:

Some piece of logic to be A/B tested
A response HTTP header that tells Fastly you're doing an A/B test
A HTML meta tag that will be used to measure the results in Google Analytics

1. Example A/B test logic

Let's say you have this controller:

# app/controllers/party_controller.rb
class PartyController < ApplicationController
  def show
    ab_test = GovukAbTesting::AbTest.new(
      "your_ab_test_name",
      allowed_variants: ['NoChange', 'LongTitle', 'ShortTitle'],
      control_variant: 'NoChange'
    )
    @requested_variant = ab_test.requested_variant(request.headers)
    @requested_variant.configure_response(response)

    case true
    when @requested_variant.variant?('LongTitle')
      render "show_template_with_long_title"
    when @requested_variant.variant?('ShortTitle')
      render "show_template_with_short_title"
    else
      render "show"
    end
  end
end

In this example, we are running a multivariate test with 3 options being tested: the existing version (control), and two title changes. The minimum number of variants in any test should be two.

2. HTTP response header to Fastly

The configure_response method used in the example in step 1 sends the response header. The header helps Fastly to understand which variant was returned to the user and cache appropriately.

3. Add HTML metatag tags to your layouts

This is for the extension and analytics.

<!-- application.html.erb -->
<head>
  <%= @requested_variant.analytics_meta_tag.html_safe %>
</head>

The analytics meta tag will include the allowed variants so the extension knows which variants to suggest to the user.

Running the test suite for the gem

bundle exec rake

Acceptance testing

Start by defining which acceptance testing framework you will use. This gem supports both Capybara and ActiveSupport. In order to configure it, add this to your test helper file:

GovukAbTesting.configure do |config|
  config.acceptance_test_framework = :capybara # or :active_support
end

If we use capybara, the gem expects page to be defined in the scope of the test cases. If we use ActiveSupport, the gem expects @request to be defined in the scope of the test cases.

Test helpers

RSpec

It is also possible to use with_variant and all the individual setup and assertions steps in RSpec tests. Here is an example of a Capybara feature file:

# spec/features/ab_testing_spec.rb
feature "Viewing a page with an A/B test" do
  include GovukAbTesting::RspecHelpers

  scenario "viewing the B version of the page" do
    with_variant your_ab_test_name: 'B' do
      visit root_path

      expect(page).to have_breadcrumbs
      expect(page).to have_beta_label
    end
  end
end

And here is an RSpec controller test:

# spec/controllers/some_controller_spec.rb
describe SomeController, type :controller do
  include GovukAbTesting::RspecHelpers

  # RSpec doesn't render views for controller specs by default
  render_views

  it "should render the B version of the page" do
    with_variant your_ab_test_name: 'B' do
      get :index
    end
  end
end

As with the minitest version, you can also pass in the following options to with_variant:

assert_meta_tag: false

Minitest

The most common usage of an A/B test is to serve two different variants of the same page. In this situation, you can test the controller using with_variant. It will configure the request and assert that the response is configured correctly:

# test/controllers/party_controller_test.rb
class PartyControllerTest < ActionController::TestCase
  include GovukAbTesting::MinitestHelpers

  should "show the user the B version" do
    with_variant your_ab_test_name: "B" do
      get :show

      # Optional assertions about page content of the B variant
    end
  end
end

Pass the assert_meta_tag: false option to skip assertions about the meta tag, for example because the variant returns a redirect response rather than returning an HTML page.

# test/controllers/party_controller_test.rb
class PartyControllerTest < ActionController::TestCase
  include GovukAbTesting::MinitestHelpers

  should "redirect the user the B version" do
    with_variant your_ab_test_name: "B", assert_meta_tag: false do
      get :show

      assert_response 302
      assert_redirected_to { controller: "other_controller", action: "show" }
    end
  end
end

To test the negative case in which a page is unaffected by the A/B test:

# test/controllers/party_controller_test.rb
class PartyControllerTest < ActionController::TestCase
  include GovukAbTesting::MinitestHelpers

  should "show the original" do
    setup_ab_variant("your_ab_test_name", "B")

    get :show

    assert_response_not_modified_for_ab_test("your_ab_test_name")
  end
end

There are some more fine-grained assertions which you can use to test a page with A/B variants which should be cached separately, but which should be excluded from the analytics:

# test/controllers/party_controller_test.rb
class PartyControllerTest < ActionController::TestCase
  include GovukAbTesting::MinitestHelpers

  should "cache each variant but not add analytics" do
    setup_ab_variant("your_ab_test_name", "B")

    get :show

    assert_response_is_cached_by_variant("your_ab_test_name")
    assert_page_not_tracked_in_ab_test("your_ab_test_name")
  end
end

API documentation

See RubyDoc for documentation including all of the assertions for tests.

To run a Yard server locally to preview documentation, run:

$ bundle exec yard server --reload

Checking your A/B test in a browser

If you want to test this behaviour in a browser then you should use the GOV.UK Toolkit browser extension.

This detects when you have a test running on a page and enables you to choose between variants.

Licence

MIT License

govuk_ab_testing

Development