pingpong

A distributed HTTP-based and REST-like ping-pong system for test and benchmarking purposes.

If you are interested in some of the details there is a research paper about ppbench.

The intended usage of this package is to run two hosts which are queried (benchmarked) by a third host (the siege). Host 1 runs a ping service querying a pong service (on host 2) for the answer. Ping and pong build a very simple distributed system communicating via a simple HTTP based resource API.

From a benchmark host (this is called the siege host) ppbench is run againt host 1. Ping host 1 has to interact with pong host 2 to answer the request. The interaction between both hosts is very simple. Whenever host 1 (ping) is asked to deliver a document for '/ping/{n}' this request is passed forward to host 2 (pong). Host 2 (pong) returns the answer which is formed of a message "pooooong" where the message is as long in bytes as the number '{nr}' provided with the query.

So we (or better ppbench) can vary the message size (and therefore the network load) between ping (host 1) and pong (host 2).

This setting shall be used to analyse the impact of infrastructures where ping and pong services are running on. The deployment above stays the same for every experiment. Just the underlying infrastructure of ping and pong changes.

• Ping and Pong might be realized in different programming languages.
• Ping and Pong might be deployed on different virtual machine types.
• Ping and Pong might be deployed on different IaaS infrastructures.
• Ping and Pong might be deployed in a containerized form.
• Ping and Pong might be connected with an overlay network.
• and so on

Therefore variations of benchmark results can be assigned to above mentioned changes in infrastructure.

To do a benchmark you have to set up a ping and a pong host. This is highly automated and will install necessary dependencies. These include:

• Dart SDK
• Docker
• Docker overlay network Weave.
• Ruby runtime and development environment
• Golang SDK
• Java SDK
• ppbench (as benchmarking and analyzing front end)

It is possible to run the Ping and Pong service as a Docker container and as a Docker container connected to a Weave SDN network. Further container solutions and SDNs are planned for further releases.

If you are working with Ubuntu LTS 14.04 cloud machines (following user-data is not tested with other distributions) capable to do cloud-config you can use the following user-data to simplify the setup of your machines a little bit:

#cloud-config

packages:
  - git
  
runcmd:
  - [git, clone, "https://github.com/nkratzke/pingpong", /home/ubuntu/pingpong]
  - [chown, "-R", "ubuntu:ubuntu", /home/ubuntu/pingpong]

After booting simply run

./install.sh

in your /home/ubuntu directory to do all the installation. This works for the pong as well as the ping host.

First step is to start the pong service on the pong host. This will start the pong service on the host on port 8080. You have three options to start the pong service:

pong:$ ./start.sh bare pong-{lang}

You want to check wether the pong service is working correctly by checking that

pong:$ curl http://localhost:8080/pong/5

answers with 'poong'.

pong:$ ./start.sh docker pong-{lang}

This will build the ping pong image if necessary. So start up may take some time. You can check whether the pong service is running:

sudo docker ps

You want to check wether the pong service is working correctly by checking that

pong:$ curl http://localhost:8080/pong/5

answers with 'poong'.

Please figure out the IP adress or DNS name of your pong host. We will refer to it as <ponghostip>.

pong:$ ./start.sh weave pong-{lang}

This will build the ping pong image as well as the necessary weave containers if necessary. So start up may take some time (longer than docker start up above). You can check whether the pong service is running:

You can check whether this container was successfully added to the weave network.

sudo weave status dns

should return something like that

pong         10.2.1.1        3efca64dc4aa 86:32:95:f5:e5:00

You want to check wether the pong service is working correctly by checking that

pong:$ curl http://localhost:8080/pong/5

answers with 'poong'.

Please figure out the IP adress or DNS name of your pong host. We will refer to it as <ponghostip>.

Second step is to start the ping service on the ping host. This will start the ping service on the host on port 8080. You will have to provide the ping service where it will find its pong service by providing <pongip> what you have figured out for the pong service above.

You have three options to do this:

pong:$ ./start.sh bare ping-{lang} <ponghostip>

You want to check wether the ping service is working correctly by checking that

ping:$ curl http://localhost:8080/ping/5

answers with 'poong'.

pong:$ ./start.sh docker ping-{lang} <ponghostip>

This will build necessary images. So startup may take some time.

You want to check wether the ping service is working correctly by checking that

ping:$ curl http://localhost:8080/ping/5

answers with 'poong'.

pong:$ ./start.sh weave ping-{lang} <ponghostip>

This will build necessary images and will connect to the SDN network established by the pong host. So startup may take some time.

You can check whether this container was successfully added to the weave network.

sudo weave status dns

should return something like that

ping         10.2.1.2        edb8527251a4 0e:37:c7:60:f2:d0
pong         10.2.1.1        2214f948aaa4 3e:07:d4:54:4e:d9

You want to check wether the ping service is working correctly by checking that

ping:$ curl http://localhost:8080/ping/5

answers with 'poong'.

We provide ppbench via RubyGems.org. So, installing ppbench on your siege system (we assume that this is your personal laptop or workstation) is very easy. Assuming you have Ruby 2.2 (or higher installed) installed, simply run

gem install ppbench

to install ppbench.

ppbench provides several commands and parameters to run and analyze your experiments. ppbench comes with an online help included. Simply run

ppbench help

to get some online help about available commands ppbench is providing.

A benchmark run is started like that:

ppbench run --host http://<pinghostip>:8080 \
            --experiment experiment_tag \
            --machine machine_tag \
            log.csv

A benchmark run can be defined via several parameters. To learn more, simply run:

ppbench help run

All benchmark results are written into a log (csv format). These csv based log files can be processed by ppbench. Ppbench is able to do some summary analysis on a set of collected benchmark files for a quick analysis. Simply run

ppbench summary *.csv

to get a result like this:

We have data for: 
+-------------+-----------+---------+-----------------+--------------+--------------+
| Experiment  | Machine   | Samples | Transfer (kB/s) | Requests/sec | Latency (ms) |
+-------------+-----------+---------+-----------------+--------------+--------------+
| bare-dart   | m3.xlarge |    4999 |        50752.03 |       298.51 |         3.35 |
+-------------+-----------+---------+-----------------+--------------+--------------+
| bare-go     | m3.xlarge |    4999 |        47329.98 |       194.53 |         5.14 |
+-------------+-----------+---------+-----------------+--------------+--------------+
| bare-java   | m3.xlarge |    2350 |        27749.37 |       111.94 |         8.93 |
+-------------+-----------+---------+-----------------+--------------+--------------+
| docker-dart | m3.xlarge |    4999 |        47892.43 |       276.13 |         3.62 |
+-------------+-----------+---------+-----------------+--------------+--------------+
| docker-go   | m3.xlarge |    4999 |        43761.69 |       183.17 |         5.46 |
+-------------+-----------+---------+-----------------+--------------+--------------+
| docker-java | m3.xlarge |    3535 |        28128.78 |       116.15 |         8.61 |
+-------------+-----------+---------+-----------------+--------------+--------------+
| weave-dart  | m3.xlarge |    4999 |        33645.47 |       189.50 |         5.28 |
+-------------+-----------+---------+-----------------+--------------+--------------+
| weave-go    | m3.xlarge |    9115 |        35385.75 |       152.34 |         6.56 |
+-------------+-----------+---------+-----------------+--------------+--------------+
| weave-java  | m3.xlarge |    4999 |        23586.90 |        92.07 |        10.86 |
+-------------+-----------+---------+-----------------+--------------+--------------+

But much more interesting and helpful (summary data is intended to be used for completenes and plausability checking of data but not for detailed analysis), ppbench is able to generate R scripts for analysis and visualization of benchmark data.

This command chain here (using Rscript and assuming you have the statistical framework R installed)

ppbench transfer-plot --machines m3.xlarge \
                      --experiments bare-dart,bare-go,bare-java \
                      --pdf graphic.pdf \
                      *.csv | Rscript -

would produce a R script which will generate a scatter plot of measured transfer rates on all benchmark runs that have been tagged to be run on m3.xlarge (AWS virtual machines) with the bare deployment of ping and pong services implemented in Dart, Go or Java programming languages. So the performance impact of several programming language to data transfer rates can be compared visually. A typical plot might look like this one here.

The plot commands of ppbench have several flags to tune your plotting. By using the following additions flags it is possible to plot 75% confidence bands without showing all measured detail data points (omitting the --nopoints would show both, confidence bands and all data points).

ppbench transfer-plot --machines m3.xlarge \
                      --experiments bare-dart,bare-go,bare-java \
                      --withbands \
                      --confidence 75 \
                      --nopoints \
                      --pdf graphic.pdf \
                      *.csv | Rscript -

This would produce a much clearer picture with additional descriptive statistical information.

Many thanks to our contributors.

• René Peinl and his Systems Integration Research Group at Hof University (Institute for Information Systems) for providing the Calico SDN integration.