FuzzBert

A random testing/fuzzer framework for Ruby.

Random Testing (or "fuzzing") is not really new, it has been around for quite some time. Yet it still hasn't found widespread adoption in everyday coding practices, much too often it is only used for the purpose of finding exploits for existing applications or libraries. FuzzBert wants to improve this situation. It's a simple fuzzing framework with an RSpec-like DSL that will allow you to integrate random tests in your project with minimal effort.

For further information on random testing, here are two excellent starting points:

Installation

gem install fuzzbert
fuzzbert --help

Defining a random test

FuzzBert defines an RSpec-like DSL that can be used to define different fuzzing scenarios. The DSL uses three words: fuzz, deploy and data.

Here is a quick example that fuzzes JSON.parse:

require 'json'
require 'fuzzbert'

fuzz "JSON.parse" do

  deploy do |data|
    begin
      JSON.parse data
    rescue StandardError
      #fine, we just want to capture crashes
    end
  end

  data "completely random" do
    FuzzBert::Generators.random
  end

  data "enclosing curly braces" do
    c = FuzzBert::Container.new
    c << FuzzBert::Generators.fixed("{")
    c << FuzzBert::Generators.random
    c << FuzzBert::Generators.fixed("}")
    c.generator
  end

  data "my custom generator" do
    prng = Random.new
    lambda do
      buf = '{ user: { '
      buf << prng.bytes(100)
      buf << ' } }'
    end
  end

end

fuzz can be thought of as defining a new scenario, such as "fuzz this command line tool", "fuzz this particular URL of my web app" or "fuzz this library method taking external input".

Within a fuzz block, there must be one occurrences of deploy and one or several occurrences of data. The deploy block is the spot where we deliver the random payload that has been generated. It is agnostic about the actual target in order to leave you free to fuzz whatever you require in your particular case. The data blocks define the shape of the random data being generated. There can be more than one such block because it is often beneficial to not only shoot completely random data at the target - you often want to deliver more structured data as well, trying to find the edge cases deeper within your code. Good random test suites make use of both - totally random data as well as structured data - in order to cover as much "code surface" as possible.

The deploy block takes the generated data as a parameter. The block itself is responsible of deploying the payload. An execution is considered successful if the deploy block passes with no uncaught error being raised. If an error slips through or if the Ruby process crashes altogether, the execution is of course considered as a failure.

data blocks must return a lambda or proc that takes no argument. You can either choose completely custom lambdas of your own or use those predefined for you in FuzzBert::Generators.

Running a random test

Once the FuzzBert files are set up, you may run your tests similar to how you would run unit tests:

fuzzbert "fuzz/**/fuzz_*.rb"

If your FuzzBert files are already in a directory named 'fuzz' and each of them begins with 'fuzz_', you may omit the pattern altogether.

Each fuzz block defines a TestSuite. These are executed in a round-robin manner. Each individual TestSuite will then apply the deploy block with a sample of data generated successively by each one of the data blocks. Once all data blocks are used up, the next TestSuite will be executed etc. By default, a FuzzBert fuzzing session runs forever, until the process is either killed or by manually hitting CTRL+C for example. This was a deliberate design choice since random testing suites need to be run for quite some time to be effective. It's something you want to run over the weekend rather than for a couple of minutes. Still, it can make sense to explicitly limit the number of runs, for example when integrating FuzzBert with a CI server or with Travis. You can do so by passing the --limit parameter to the fuzzbert executable:

fuzzbert --limit 1000000 "fuzz/**/fuzz_*.rb"

Every single execution of deploy is run in a separate process. The main reason for this is that we typically want to detect hard crashes when a C extension or even Ruby itself encounters an input it can't handle. Besides being able to cope with these cases, running in separate processes proves beneficial otherwise as well: by default, FuzzBert runs the tests in four separate processes at once, therefore utilizing your CPU's cores effectively. You can tweak that setting with --pool-size to set this number to 1 (for completely sequential runs) or to the exact number of cores your CPU offers.

fuzzbert --pool-size 1 my/fuzzbert/file

What happens if we encounter a bug?

If a test should end up failing (either the process crashed completely or caused an uncaught error), FuzzBert will output the failing test on your terminal and tell you where it stored the data that caused this. This conveniently allows you to run FuzzBert over the weekend and when you return on Monday, the troubleshooters will sit there all lined up for you to go through and filter. By using the --console command line switch you can tell FuzzBert to not explicitly store the data, but echoing the data that caused the crash to the terminal instead.

fuzzbert --console "fuzz/**/fuzz_*.rb"

If you don't want to litter your current working directory with the files generated by FuzzBert, you can also specify a specific path to where they should be saved instead:

fuzzbert --bug-dir bugs "fuzz/**/fuzz_*.rb"

This is still not quite what you want to happen in case a test crashes? There's also the possibility to define a handler of your own:

require 'fuzzbert'

class MyHandler
  def handle(error_data)
    #create an issue in the bug tracker
    puts error_data[:id]
    p error_data[:data]
    puts error_data[:pid]
    puts error_data[:status]
  end
end

fuzz "Define here as usual" do
  ...
end

Now you just need to tell FuzzBert to use your custom handler:

fuzzbert --handler MyHandler my/fuzzbert/file

Templates

Using the approach described so far is most useful for binary protocols, but as soon as you work with mainly String-based data this can quickly become a chore. What you actually want in these situations is some sort of template mechanism that comes with mostly fixed data and only replaces a few selected parts with randomly generated data. This, too, is possible with FuzzBert, it comes with a minimal templating language:

require 'fuzzbert'

fuzz "My Web App" do

  deploy do |data|
    # Send the data to your web app with httpclient or similar.
    # You define the "error conditions": if a response to some
    # data is not as expected, you could simply raise an error
    # here.
  end

  data "JSON generated from a template" do
    t = FuzzBert::Template.new '{ user: { id: ${id}, name: "${name}" } }'
    t.set(:id, FuzzBert::Generators.cycle(1..10000))
    t.set(:name) { "Fixed text plus two random bytes: #{FuzzBert::Generators.random_fixlen(2).call}" }
    t.generator
  end

end

Simply specify your template variables using ${..} and assign a callback for them via set. Of course you may escape the dollar sign with a backslash as usual. The Template#set method takes as its first argument the symbol representing the template variable. To specify the data to be generated, it either takes a second argument in form of a proc or lambda (to use the built-in generators for example, as in :id in the example) or it takes a block that allows to define the data ad-hoc (as in :name).

Mutators

Mutation is the principle used in "Babysitting an Army of Monkeys". The basis for the mutation tests is a valid sample of input that is then modified in exactly one position in each test instance. You can apply this principle as follows:

require 'fuzzbert'

fuzz "Web App" do
  deploy do |data|
    #send JSON data via HTTP
  end

  data "mutated data" do
    m = FuzzBert::Mutator.new '{ user: { id: 42, name: "FuzzBert" }'
    m.generator
  end

end

This will take the original JSON data and modify one byte each time data is being generated.

Rake integration

You may integrate Rake tasks for FuzzBert similar to how you would include a task for Rspec:

require 'rake'
require 'fuzzbert/rake_task'

FuzzBert::RakeTask.new(:fuzz) do |spec|
  spec.fuzzbert_opts = ['--limit 10000000', '--console']
  spec.pattern = 'fuzz/**/fuzz_*.rb'
end

Supported versions

FuzzBert has been confirmed to run on CRuby 1.9.3 and Rubinius 2.0.0dev. Since it heavily relies on forking, it does not run on JRuby so far, but support is planned and on its way.

You may also use FuzzBert for fuzzing arbitrary applications or libraries that aren't connected to Ruby at all - have a look in the examples that ship with FuzzBert.

fuzzbert