Paraspec

Paraspec is a parallel RSpec test runner.

It is built with a producer/consumer architecture. A master process loads the entire test suite and sets up a queue to feed the tests to the workers. Each worker requests a test from the master, runs it, reports the results back to the master and requests the next test until there are no more left.

This producer/consumer architecture enables a number of features:

1. The worker load is naturally balanced. If a worker happens to come across a slow test, the other workers keep chugging away at the remaining tests.
2. Tests defined in a single file can be executed by multiple workers, since paraspec operates on a test by test basis and not on a file by file basis.
3. Standard output and error streams can be[*] captured and grouped on a test by test basis, avoiding interleaving output of different tests together. This output capture can be performed for output generated by C extensions as well as plain Ruby code.
4. Test results are seamlessly integrated by the master, such that a parallel run produces a single progress bar with Fuubar across all workers.
5. Test grouping: paraspec can[**] group the tests by file, top level example group, bottom level example group and individually by examples. Larger groupings reduce overhead but limit concurrency. Smaller groupings have more overhead but higher concurrency.
6. Paraspec is naturally resilient to worker failure. If a worker dies for any reason the remaining tests get automatically redistributed among the remaining workers. It is also possible, in theory, to provision additional workers when a test run is already in progress, though this feature is not currently on the roadmap.

[*] This feature is not yet implemented.

[**] Currently only grouping by bottom level example group is implemented.

How much of a difference does paraspec make? The answer, as one might expect, varies greatly with the test suite being run as well as available hardware. Here are some examples:

Exampe Travis build

Even on Travis, which is likely limited to a single core, using 4x concurrency reduces the runtime by 5 minutes. On a developer workstation which doesn't download binaries on every test run the speedup is closer to linear. Waiting 4 minutes instead of 15 for a complete test suite means the engineers can actually run the complete test suite as part of their normal workflow, instead of sending the code to a CI platform and context switching to a different project.

Add paraspec to your Gemfile:

gem 'paraspec'

This is necessary because paraspec has its own dependencies and loads the application being tested into its environment, hence both paraspec's and application's dependencies need to exist in the same Bundler environment.

Then, for a test suite with no external dependencies, using paraspec is trivially easy. Just run:

paraspec

To specify concurrency manually:

paraspec -c 4

To pass options to rspec, for example to filter examples to run:

paraspec -- -e 'My test'
paraspec -- spec/my_spec.rb

For a test suite with external dependencies, paraspec sets the TEST_ENV_NUMBER environment variable to an integer starting from 1 corresponding to the worker number, like parallel_tests does. The test suite can then configure itself differently in each worker.

By default the master process doesn't have TEST_ENV_NUMBER set. To have that set to 1 use --master-is-1 option to paraspec:

paraspec --master-is-1

Paraspec works with any RSpec formatter, and supports multiple formatters just like RSpec does. If your test suite is big enough for parallel execution to make a difference, chances are the default progress and documentation formatters aren't too useful for dealing with its output.

I recommend Fuubar and RSpec JUnit Formatter configured at the same time. Fuubar produces a very nice looking progress bar plus it prints failures and exceptions to the terminal as soon as they occur. JUnit output, passed through a JUnit XML to HTML converter like junit2html, is much handier than going through terminal output when a run produces 100 or 1000 failing tests.

Paraspec offers several debugging aids. For interactive debugging use the terminal option:

paraspec -T

This option makes paraspec stay attached to the terminal it was launched in, making it possible to insert e.g. byebug calls in supervisor, master or worker code as well as anywhere in the test suite being executed and have byebug work. Setting this option also removes internal timeouts on interprocess waits and sets concurrency to 1, however concurrency can be reset with a subsequent -c option:

paraspec -T -c 2

Paraspec can produce copious debugging output in several facilities. The debugging output is turned on with -d/--debug option:

paraspec -d state   # supervisor, master, worker state transitions
paraspec -d ipc     # IPC requests and responses
paraspec -d perf    # timing & performance information

It is possible to specify that a group of tests should be executed in the same worker rather than distributed. This is useful when the setup for the tests is expensive and therefore is done once with multiple tests defined on the result.

The grouping is defined by specifying {group: true} paraspec option on a describe or a context block as follows:

describe 'Run these together', paraspec: {group: true} do
  before(:all) do
    # expensive setup
  end
  
  it 'does something' do
    # ...
  end
  
  it 'does something else' do
    # ...
  end
  
  after(:all) do
    # teardown
  end
end

The master and workers need to all define the same example groups and examples, otherwise a worker may retrieve an example group from the master that it is unable to run. RSpec provides a way to define tests conditionally via if: and unless: options on contexts and examples, and if this functionality is used it is important to make sure that all workers are configured identically (especially if such conditional configuration is dependent on external resources such as a network server).

Paraspec will check workers' example groups and examples for equality with master and will raise an error if there is a mismatch.

Aborting a test run with Ctrl-C can leave the workers still running.

Please report via issues and pull requests.

MIT

Knapsack