Skip to content

Writing a Real C++ Program – Part 10

September 2, 2011

This is the tenth instalment in a series of C++ programming tutorials that started here.

Introduction

In this instalment, I would like to address some of the issues surrounding unit testing. Some experienced readers may think that this is not before time, and I would agree that unit testing of should take place before its presentation in this series would suggest. Please think of this as actually being instalment number five.

Unit Testing

Unit testing is the testing of, well … units! Which begs the the question: what is a unit? The answer to this will vary somewhat between computer languages (and with whom you are talking to) but in C++ the basic unit is the function, so unit tests for C++ programs are tests that run at the level of single functions.

The main purpose of unit testing is to give you confidence that your code works, and more importantly, to give you confidence that you can change it and have any errors you introduce brought immediately to your attention. This is extremely important; there are many test-less applications out there who’s developers are frankly too scared to change anything, and you really don’t want want to be in that state. If you have good tests in place, you can make quite major changes, run all your tests, and (if the tests pass) have a high confidence that your changes haven’t broken anything.

As an example of a unit test, let’s consider the NextWord function that is a member of the scheck Parser class. If you remember, this pulls the next word out of a submission (represented by an istream), ignores numbers and punctuation, and returns an empty string when all words have been extracted. What kind of unit test might you want to write for this function?

Unit tests (and tests in general) should always primarily test that the common case works. So in this case, you would give the parser input containing some words, have it extract them, and then test that the words extracted match those that form the input. In pseudocode, you want something like this:

set up the parser with input "quick brown fox"
extract first word and check it is "quick"
extract second word and check it is "brown"
extract third word and check it is "fox"
extract fourth word and check it is the empty string

This tests quite a lot of the parser (but not all, you’ll see other tests are needed), now where to put this code? It should not go in the Parser class itself, as you do not want your test code obscuring or interfering with the business logic of your application. Ideally, you want the test code to be developed alongside the application code, but be clearly separated from it. The normal course is to use a unit test framework which enforces the separation, but makes calling the test code easy.

The MiniTest Framework

There are many unit test frameworks for C++, for example cppunit and and the Boost testing framework. They all seem to have one thing in common, and that is that they are all somewhat over-engineered. Although I would encourage you to try out several of the available frameworks until you find the one you like, I’m going to present  a hyper-simple framework of my own invention, which illustrates most of the techniques that the more complex frameworks use. It’s implemented in the minitest.h header which you can find  in the sources for this instalment – link at the bottom of this blog post.

To write the test described above,  add the following code to the end of the parser.cpp source code:

#include "minitest.h"
#include <sstream>

MINI_TEST( parser_words1 ) {
  std::istringstream is( "quick brown fox" );	
  Parser p( is );
  MINI_CHECK( p.NextWord() == "quick" );
  MINI_CHECK( p.NextWord() == "brown" );
  MINI_CHECK( p.NextWord() == "fox" );
  MINI_CHECK( p.NextWord() == "" );
}

The body of the test is just standard C++ code, and in fact each test is a really a single C++ function.

A point about the naming of the test – names are really not important, because mostly they all pass and you don’t care. Occasionally a test will fail, and as you will see the framework tells you exactly where. So don’t agonise over "meaningful" test names. This is true for all unit test frameworks, not just MiniTest, though a limitation of MiniTest is that the test names must be valid C++ identifiers.

You will also need to change your main.cpp source slightly. You need to include the minitest.h header there too, and then change the main function so it looks like this:

int main( int argc, char * argv[] ) {
  MINI_MAIN();
  /* rest as before */
}

If you now rebuild the scheck application using the makefile and then run it, you should get this:

$ bin/scheck.exe
parser_words1 ...  passed

The framework has run the test for you, and told you that it has passed. I’ll explain how it does this a bit later, but try adding another test, this time for numbers. Remember, numbers (or words containing digits) don’t get returned by NextWord. Add this test to what you already have:

MINI_TEST( parser_numbers1 ) {
  std::istringstream is( "quick 42 lucky7 fox" );	
  Parser p( is );
  MINI_CHECK( p.NextWord() == "quick" );
  MINI_CHECK( p.NextWord() == "fox" );
  MINI_CHECK( p.NextWord() == "" );
}

Rebuild and run scheck again:

$ bin/scheck.exe
parser_numbers1 ...  passed
parser_words1 ...  passed

Ok, but what happens when a test fails? Try writing a test that should fail:

MINI_TEST( parser_wrongfox ) {
  std::istringstream is( "fox" );	
  Parser p( is );
  MINI_CHECK( p.NextWord() == "dog" );
}

Running this gives:

$ bin/scheck.exe
parser_numbers1 ...  passed
parser_words1 ...  passed
parser_wrongfox ...  failed [ p.NextWord() == "dog" ] is false in src/parser.cpp at line 124

The failed test is identified, together with the condition that caused the failure, the file name containing the test, and the line number in the file of the use of MINI_CHECK that caused the failure.

What To Test?

OK, so you now have a means of writing simple tests alongside your code – but what to test? There are lots of opinions on this, but here’s my advice:

  • Make sure you test the common case first. In the case of the Parser, that test is "can I extract words from text correctly?" This kind of test is simple to understand and write, so do it first.
  • Test edge cases. If I were writing a mathematical function, I would test specifically that it worked for the numbers 0, 1 and -1 (assuming it was defined for such numbers). For the Parser, I’d test that it worked for empty strings, and strings containing only whitespace or punctuation.
  • Test the things you don’t feel so confident about. In the case of Parser, I would test the hyphenation code and the stuff that handles apostrophes (there are still bugs in there – can you find them?)
  • Test the public interface. Writing tests that can be used on private implentation functions usually involves a bit of hoop-jumping, so don’t do it unless you have to. also, private functions tend to change a lot, which implies that your test code will have to be rewritten too.

The things I would not test include:

  • External inputs. For example, in the case of the Parser class I would not test that the stream that gets passed to its constructor is open. That test belongs (if it is required at all) in the unit(s) that use the Parser.
  • External systems. If your class uses external systems or resources such as databases or files, I would put off testing the use of these until the application testing level, which I’ll write about in the next instalment.

Another word of advice, keep the tests simple, and don’t go mad with them. If you find you have written 10 times as much testing as application code, something has gone wrong somewhere, in my opinion.

When To Write And Run Tests?

Adherents of Test Driven Design (TDD) would have you believe that tests should be written before you write the code they test. This is an interesting idea, but one that does not not work too well for new functionality, in my experience. Instead, I prefer to write my first tests after I have completed the code first pass of the code for a particular function in a unit. As I write more functions (or discover problems with the functions I’ve already written), I write more tests.

When to run the tests is easy – you run them after every compilation of your code, and they must all pass. If a test fails you must fix the problem it highlights  immediately – do not allow any failed tests. I cannot stress this strongly enough – your code should always be at a zero-failure state when it comes to unit tests.

If you take the zero-failures rule seriously, then if bug is reported in your application it implies that you have not got a test for that particular corner of the code. When this happens you should stop whatever you are doing and write a test that fails and thus highlights this particular bug. Do that before you try to fix the bug.  You will now be in a non zero-failure state, which means you must stop everything until you have got the new test to pass.

If you follow this rule, you will find that your code will become remarkably bug free, to the extent you can forget about the use of tools like bug-trackers. You will, as I suggested earlier, also be able to make large changes to the code with confidence.

How Does It Work

Here I’ll give a brief explanation of how the MiniTest framework is implemented – you can skip this if you are not interested, but it does use some C++ features that you might find interesting to use in your own code.

You’ve already seen that MiniTest is implemented using three main preprocessor macros, all of which are defined (along with all the rest of the MiniTest source code) in minitest.h. By the way, don’t believe all you read about macros being "evil"; they are simply tools that you use when appropriate. The first macro, MINI_MAIN simply expands to this code:

MiniTest::Tester::Instance().RunTests( );  
return MiniTest::Tester::Instance().Errors();

This simply calls a static member variable Instance on the MiniTest::Tester class, and then calls a couple of functions on the return value of it. The Instance function looks like this:

static Tester & Instance() {
  static Tester instance;
  return instance;
}

The first time this function is called, the static Tester object is created – that’s how function-level statics work in both C and C++. This only happens once, and from then on the static Tester object exists until the program terminates. The function simply returns a reference to this object. This is in fact the only way of creating a Tester object, as its constructor has been declared private. Classes like this, which only allow the creation of a single instance of the class, are called singletons. Like macros, singletons often get a bad press, but like macros they are simply tools one uses when appropriate.

So, the MINI_MAIN macro simply calls two functions RunTests and Errors on the singleton instance of Tester. The RunTests function runs all the tests that you have created with the MINI_TEST macro, and the the Errors function returns a count of all of the failures of the MINI_CHECK macro in those tests.  But how does the Tester know about the tests that you wrote?

The MINI_TEST macro looks like this:

#define MINI_TEST( testname ) \
  static void testname();\
  static MiniTest::TestAdder TA_##testname( #testname, testname );\
  void testname() 				

Let’s write a pretty useless test using it:

MINI_TEST( silly ) {
  int x = 0;
}
				

This will be expanded by the pre-processor into:

static void silly();						
static MiniTest::TestAdder TA_silly( "silly", silly ); 
void silly() {
  int x = 0;
}

This declares the silly test function as static, which limits its scope to the file, avoiding you having to think up globally unique names for tests. It then creates a static instance of  TestAdder class, passing a string representation of the function name and the actual function name as parameters. Then there follows the test function code itself.

The key thing here is the static TestAdder instance. Objects declared as static at file scope in C++ are created for you automatically by the compiler before the main() function is entered. If they have constructors, the constructor code will run before any code in main is executed. In this case, the constructor code is as follows:

TestAdder( const string & name, TestFn fn ) {
  Tester::Instance().AddTest( name, fn );
}

which simply adds the name of a test and a function pointer to a test to the singleton Tester instance, where they are stored in a map. Every test function you write using the MINI_TEST macro will have  an associated TestAdder object which is responsible for adding the test to Tester instance.

The final bit of the jigsaw is the MINI_CHECK macro, which is declared like this:

#define MINI_CHECK( cond )	
  if ( ! (cond) ) { \
    throw Error( #cond, __FILE__, __LINE__ );\
  }

If your test contained this check:

MINI_CHECK( x == 0 )

it would be expanded into something like this:

if ( ! ( x == 0 ) ) {						
  throw Error( "x == 0", "parser.cpp", 87 );
}

which just constructs an Error object from a string representation of the condition, the file containing the macro (provided by the built-in __FILE__ macro) and the line number in the file (provided by the built-in __LINE__ macro).

So, all the Tester::RunTests function that MINI_MAIN provides does is to walk through the functions in its map that were added by the MINI_TEST macro, execute them and catch any Error exceptions thrown, using the values provided by MINI_CHECK to construct a suitable error message.

Most unit test frameworks use something like this simple solution to register and run their tests. It does have a couple of drawbacks, the main one being that it will not work for static libraries – I’ll cover this when I talk about building and using libraries.

Conclusion

Unit testing is a powerful tool for making good software and for making you feel good about the software you produce. In the next instalment, I look at a couple more testing techniques you should use to maintain that good feeling.

Sources for this and all other tutorials in the series are available here.

About these ads

From → c++, linux, tutorial, windows

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: