Project

greybox

0.0
No commit activity in last 3 years
No release in over 3 years
Test against a black box.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

~> 1.3
>= 0
>= 0
>= 0

Runtime

>= 0
 Project Readme

Build Status Code Climate

Greybox

Test a CLI program against a black box. Intended for school projects.

Often when working on a school project, the class will provide a black box to test your programs against. Greybox provides a simple testing harness to easily add new test cases whose expected output is automatically generated by this black box. Since these might be slow, or even over an ssh connection, it will store the results so it only has to actually run the black box the first time.

Installation

Only Ruby 1.9.3 and 2.0.0 are supported.

$ gem install greybox

Usage

In your project directory, create a file called Greyfile. A Greyfile might look like this:

Greybox.setup do |c|
  c.test_command = "./my_test_command < %"
  c.blackbox_command = "./blackbox_command < %"
  c.input = "test/*.input"
end

This defines a testing setup in which the command

$ ./blackbox_command < INPUT_FILE.input

is used as the reference for correctness. % denotes where the input file will be placed.

$ ./my_test_command < INPUT_FILE.input

is verified to match the blackbox output every time

$ greybox

is run.

The expected results will be stored in INPUT_FILE.output. This is the default behaviour but can be changed.

As a more complete example, here is a Greyfile I used for a school project:

Greybox.setup do |c|
  c.test_command = "racket wlppscan.ss < % 2>&1"
  c.blackbox_command = "ssh school 'java cs241.WLPPScan' < % 2>&1"
  c.input = "test/*.input"

  # if the expected ERRORs, all we care about is that
  # the actual ERRORs as well.
  c.comparison = ->(actual, expected) do
    if expected =~ /ERROR/
      actual =~ /ERROR/
    else
      expected == actual
    end
  end
end

The test_command specifies how to run my version of the program, a racket program. blackbox_command specifies how to generate the expected output for a specific test case.

comparison specifies how the output of test_command should be compared to the black box output. In this case, if part of the output contains ERROR, all we care about is that the actual output contains ERROR as well. Otherwise, they must be equal.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request