CS35 Lab 2: Automated Testing

Due 11:59pm Sunday, September 13, 2014

The goal of this lab is to introduce you to automated testing, and to think about testing the correctness of a simple data structure. You'll see:

To fetch the skeleton for this assignment run:

$ git clone git@github.swarthmore.edu:cs35-f15/lab2-<your-username>.git
A skeleton version the program will appear in the lab2-<your-username> directory when you run this. This is a solo lab.
Automated Testing

For anything beyond the simplest programs, manual testing—by typing in the inputs and looking at the results that come back—is tedious and quickly becomes impractical. Every time a change is made, the manual inputs need to be re-tested, and it’s easy to make a typo that derails the entire testing effort, forcing the tester to redo work. For these reasons, as software gets more complicated, it becomes necessary to test it automatically.

There are a number of strategies for automated testing with all kinds of names and tradeoffs – “integration testing”, “unit testing”, “end-to-end testing”, and more. A discussion of the full spread of testing strategies is beyond the scope of CS35, so we will keep things straightforward in this course. We’ll use an open-source library called UnitTest++ that lets us check for simple properties and reports which tests failed in a summary report.

If you want to see the whole range of checks provided by UnitTest++, you can visit the Documentation. For this assignment, you will only need two check forms: CHECK_EQUAL and CHECK, along with the TEST form that organizes related checks together. There are examples of using TEST and CHECK_EQUAL in the starter files, and we summarize their use here.


A set is a simple data type that holds an unordered collection of elements. In this assignment, we'll just be talking about sets of ints. We can use sets by creating empty sets with the Set constructor (we’ll talk more about constructors in lecture; for now treat Set just like a function that constructs an object, as in Python or Java).

Set s = Set();

We can add integers to the set with the add method:


We can check for their presence using the member method:

s.member(37); // returns true
s.member(63); // returns true
s.member(52); // returns false

We can get their size using the size method:

s.size(); // returns 2

We can remove elements using the remove method:

s.remove(37); // removes 37
s.member(37); // returns false now
s.member(63); // returns true still
s.size(); // is 1 now
Finally, sets support intersection and union via intersect_with and union_with:
Set s1 = Set();
Set s2 = Set();
// now s1 contains 37, 5, and 63 (s2 is unchanged)
Set s3 = Set();
// now s1 contains only 5 (s3 is unchanged)

A summary of this behavior is included in the header file sets.h. That file looks like:

class Set {

  void add(int element);
  void remove(int element);
  bool member(int element);
  int size();

  void intersect_with(Set &other);
  void union_with(Set &other);

  struct SetRep;
  SetRep* setrep;

This describes the Set datatype (implemented as a class), but doesn't contain any function bodies for the methods that are declared, like add or member. Those have all been defined for you, but not visible to you, in the files in implementations/.

Recall from lab that the files in implementations/ are called object files, and they are compiled versions of .cpp files. They were generated by running the compiler with the -c option:

> clang++ -c set1.cpp
which builds the object file for later use. Some other call to the compiler will use the object file to provide some definitions that are declared elsewhere (like in a header file). In the case of this assignment, the test_set.cpp file expects some definitions that match what's in the header file. The Makefile links the tests against each particular incorrect implementation, each of which produces a runnable file (a binary) that produces the test results when executed.

Don’t worry if these build details don’t make complete sense to you yet; we’re providing instructions on how to run everything. Mainly I’m describing all this to start to help you develop some intuition by example for how the build process of larger C++ projects work.

Testing Task

The starter lab repository contains a number of implementations of sets, one of which is correct, with the others intentionally broken. Your task for this lab is to write a suite of tests where:

  1. All the tests pass on the correct implementation
  2. Some test fails on each incorrect implementation

Further, we will provide short descriptions of how the set implementations are broken, but we won't tell you which implementation is broken in which way (and we won't tell you which one is the correct one!). Along with your submission, you will submit a file matches.txt that matches each set implementation number with the description of its behavior, including which implementation is the correct one. Your file should look like (assuming set3 is correct, set2 is no_negatives, etc):

set1: bad_empty_size
set2: no_negatives
set3: correct
set4: bad_empty_remove
set5: bad_add_removes_sometimes
set6: bad_max_100_elements
... all the way through set16

You will add the tests in the test_sets.cpp file (there is an example to get you started). To run the tests, you can do one of a few things:

For example, the starter file you've been given has tests that pass on all the implementations, so you have some work to do. When we run make test_all at the start, we'll see something like:

# lots of build mumbo-jumbo the first time

mint:lab2$ make test_all

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

Success: 2 tests passed.
Test time: 0.00 seconds.

As tests start to fail (and you detect bad solutions), you'll start to see output like:

test_set.cpp:55:1: error: Failure in my_test_name: Expected 0 but was 1
FAILURE: 1 out of 15 tests failed (1 failures).
Test time: 0.00 seconds.

Output like this means the test at line 55 failed on the set8 implementation. As long as you're confident your test is correct, this means that your test suite has detected something wrong with the set8 implementation. The test you wrote should give you a clue about what's going on, and you can use that information to fill in which problem set8 has in matches.txt.

Implementation Descriptions

Each implementation has a short title that you should use in matches.txt to indicate which implementation number matches which description.

Handing In

You should edit only matches.txt and test_set.cpp. Add, commit, and push your changes to Github by 11:59PM on Sunday, September 13.