rspec-trace-formatter

An RSpec formatter for constructing trace data from your specs. This library is inspired by go-test-trace.

Collecting data from your RSpec tests may be useful to you for a number of reasons:

1. You'd like to see statistical trends in test runtimes, or in your CI/CD pipeline as a whole
2. You'd like a dataset containing pass/fail statuses for all tests to help hunt down flakes
3. Other things that I can't think of

Traces aren't the only choice for collecting this data, but they are a reasonable one. With concepts like test files and example groups, test execution naturally maps onto a trace tree. The flexibility of tools like OpenTelemetry when it comes to including arbitrary key-value attribute pairings is useful when instrumenting a library like RSpec because we can preserve as much context about the tests as we like. And this test data isn't likely to be valuable long-term, so the standard retention periods for traces are likely to be acceptable.

There are three main parts to this library.

RSpec::Trace::Formatter is an RSpec formatter that emits events containing data that's relevant for constructing traces. This formatter doesn't create traces -- it only outputs JSON events.

This formatter emits events for the most significant lifecycle events in an RSpec suite: the start of the suite, the start/end of each example and example group, and the end of the suite. Because all events are timestamped, you can expect accurate timing data. It also collects data about the names of the examples and example groups that are run, as well as useful facts like file locations, pass/fail status, &etc.

The event format is designed to be redundant when providing facts about examples and example groups, so as to be less prescriptive about how you consume them. This may not be the best decision, but it seemed the right way.

rspec-trace-consumer is a script that reads events created by RSpec::Trace::Formatter from standard input and emits trace data to an OpenTelemetry collector. The OpenTelemetry SDK can be configured using the standard OTEL_* environment variables.

If not set by the OTEL_SERVICE_NAME environment variable, the service name will be set to rspec. The name of the root span defaults to "rspec", but you can change that as well with the RSPEC_TRACE_FORMATTER_ROOT_SPAN_NAME environment variable.

This script uses the AlwaysOn sampler to ensure that no data is ever discarded.

RSpec::Trace::OpenTelemetryFormatter is a separate RSpec formatter that combines the two pieces above to collect trace events from RSpec tests and send them to an OpenTelemetry collector.

Because this uses the (very nice!) subprocess library, it only works on Ruby platforms where fork is supported. If you're running in an environment where this isn't supported (e.g. JRuby) you won't be able to use this. However, the rest of this library is expected to work for you, and specifying an --out target for RSpec::Trace::Formatter may make this easier.

You can install this gem by adding the rspec-trace-formatter (along with the necessary OpenTelemetry dependencies, if they aren't already included) to your Gemfile and running bundle install. For example:

group :test do
  gem "rspec-trace-formatter"
  gem "opentelemetry-api", "~> 1.0"
  gem "opentelemetry-exporter-otlp", "~> 0.20.0"
end

This library should be used like any other RSpec formatter, with the assistance of any environment variables that you need to control the OpenTelemetry data.

Example of using the RSpec::Trace::OpenTelemetryFormatter with representative environment variables set:

$ OTEL_TRACES_EXPORTER=console bundle exec rspec --format RSpec::Trace::OpenTelemetryFormatter

Example of running the RSpec::Trace::Formatter by itself and sending the output to rspec-trace-consumer separately (in a way that you can surely improve upon):

$ OTEL_TRACES_EXPORTER=console bundle exec rspec --format RSpec::Trace::Formatter --out /tmp/trace-events.jsonl

# Piping the input in
$ rspec-trace-consumer < /tmp/trace-events.jsonl

# Passing a filename as an argument
$ rspec-trace-consumer /tmp/trace-events.jsonl

If the TRACEPARENT environment variable is set in either of these cases, it will be interpreted as a W3C Trace Context Traceparent Header value. This will allow you to include the span events generated by this library in a larger distributed trace.

Very carefully, I hope.

One notable fact is that we use snapshot testing for the class underpinning rspec-trace-consumer. To keep this reliable, I've defined a custom OpenTelemetry span exporter that includes meaningful-enough data to test with and no execution-specific fields.

• rake build: Build the gem
• rake install: Builds and installs the gem
• rake regenerate_examples: Rebuilds fixtures for snapshot tests
• rake test: Runs the automated tests (written with RSpec, of course)
• rake update_snapshots: Updates the test snapshots

Configuration for a dev container is provided for convenience. The main practical benefit of developing in the container is to be able to regenerate the snapshots for the Consumer tests with consistent and non-identifying file paths for the stack traces.

MIT