Personal Blogs

Tim dumped a huge book about unit testing on my desk today. He thinks I should read some of it!

But I'd been thinking of writing a blog post about unit testing, so why not take the opportunity before I read too much of the big book and it spoils me? I'll try to write a blog post that's nearly as long as the book!

First if anyone doesn't know what unit testing is, (a) it's a technique for writing computer program code that tests other pieces of computer program code in an automatic, repeatable way so that if you change the code that's being tested, and the change breaks something, the test will probably start to fail and you'll notice... and (b) you probably don't care about this post. Go have fun somewhere else! This is about boring programming stuff.

In my recent personal projects I do significant unit testing, but I don't really follow any of the buzzwords. Here's what I think, divided into neat pairs (spot the odd one out).

Be careful - I'm going to say that you 'should' do something or that you 'must' do something else or that 'nobody' would be stupid enough to even go within bargepole-reach of the general concept of a third something. If that's likely to concern you, please just pretend I said 'in my opinion' several times per sentence.

Is unit testing worthwhile?

• Unit testing is really, really important when writing complex backend code. Unit testing won't work for UI code. This means you should ensure that as much code as possible is in the backend. But you know that already.
  Without unit tests you cannot change anything without being desperately panicky about whether it'll break (and it probably will). If you're doing something complicated you definitely want unit tests.
  JUnit is integrated into Eclipse, so if you're a Java programmer, it's easy to get started and run tests. No need to create test suites or anything. Just right-click on a source folder containing Java files with JUnit tests and 'Run As... JUnit Test' to run 'em all.
• If you're doing something simple, writing unit tests is a waste of time. Don't bother. Unit testing set/get pairs - or anything that basically boils down to that - isn't big and it isn't clever. Because Java's a compiled language and because modern compilers give warnings for things like assigning variables to themselves, the chances you would break any code that simple are pretty damn low.

How completely should you test? At what level should you test?

• When testing complex code you've written, coverage is important. Test coverage means the instructions that were covered by all your test cases. You hit run on all your test cases and something tells you what percentage of code was covered.
  If a branch wasn't covered by the test cases, it's probably a good bet that it's broken, so getting good coverage is important. I'd guess you want to be at least 80% or so, with maybe just some error cases and set/get and suchlike omitted. (If the errors are really important, i.e. might actually happen for somebody whose machine isn't already terminally broken, then you should make those tested too.)
  For Java developers, coverage testing is easy. Just install the EclEmma Eclipse plugin. It'll show you coverage by package tree, and you can look at source files to see lines that were covered (green) and weren't (red).
• So I think coverage is important. But I don't think it matters a damn where that comes from. You do not need a matching test class for every real class. If you write low-level classes which are used by higher-level ones, then fine - just do tests for the higher-level ones, provided that gives reasonable coverage (and finesse the inputs to the higher-level code so it does). Or do lower-level tests for the worst most fiendish parts of that code, but don't bother with the finicky details; leave that for the higher-level stuff that will use it.
  Some people argue this means you can't immediately tell what failed. They're right, but nobody cares. Unless you're writing the world's most complicated project, it's unlikely that a failure in a high-level test would be more than a matter of minutes to track down at a lower level. You have a built-in reproducible example right there, and you know how to use the debugger. THERE IS NO STEP THREE. Come on. (Plus, you have the clue it's probably related to, oh I dunno, maybe something in the checkin that broke the unit tests?)

What about writing test code itself?

• Test code should be well-organised so that we know which class you're testing...
• ...but it doesn't have to be good quality code. Who cares? It's test code. Stick some comments in if you're doing something non-obvious or if you feel like it but if you have a function called testFreeble and your system has a feature called freebles and you run the freeble() function 3 times inside that test... I think we know what you're testing. If you call a function with -1 or 0 then it's probably an out of bounds or edge case. We get it.
  Similarly, refactor stuff (using the automatic 'extract method' or manual refactoring) to make your life easier writing and updating tests, but not purely for code quality reasons. Nobody needs to read tests unless they break. Unless your 'real' code is already perfect, spend time improving that before you waste it on test cases, which are usually relatively simple.
  Maybe you've configured your Java compiler to give a warning if you write a public method without javadoc (I do) - you can turn this off in test code because all test methods are public and those 'Tests freebles' comments get really old really fast. @SuppressWarnings("all").

Do you write tests before or after development?

• After. Puh-leeze. You know you're going to change the API. How boring would a project be if you knew what the API was going to be before you started? And unit tests, achieving coverage, are not black-box tests; your test manipulates a black-box, but when writing the test, you choose test cases based on what you know's inside. It's much easier to do that when you've written it.
• But, ideally, straight after. Write the complicated back-end code for the project first. You haven't got a user interface to run it - so until your project's very near finished, literally the only way you should ever be running it is through hitting 'play' on the JUnit view.
  Once you finish a feature, do the test for it. Running that test, you know your code won't work, because it's the first time it's ever been run. Yay! That's a good thing. Because writing tests is much less tedious if there's some realistic chance, like in this case about 90%, that they actually fail initially, and you have to go off and fix the bugs in your actual code.

Testing in Java is a dream; 'existing horrible project' aside, there's really little excuse not to do it. Write everything complicated as a separate backend package and test that sucker before you glue it into a real projet. Everything runs at lightning speed, you've got all the nice, pretty tools built into the IDE you already use (by the way, if you prefer NetBeans, the tools are available in that too).

Sooo.. at work I mainly have to develop things for (our customisation of) Moodle, which is written in PHP. Aaaand we're off to a flying disaster right there...

Moodle uses SimpleTest, which is a perfectly okay framework, but suddenly I have to go to a webpage to run the tests instead of hitting Command-F11. At present there isn't coverage stuff in it, either. We can never get the debugger to work (that's okay, I can live with print_object). But worst of all, everything in Moodle is tightly integrated, and (this part isn't really Moodle's fault) most of the complicated bits which are likely to break are actually database queries. Which means you can't usefully test it by abstracting out the database, even if that wasn't an absolute nightmare that wastes hours of development time (and yes I've done it).

Moodle 2 provides some things to make testing fractionally easier but overall it's still a nightmare and will remain so until there's some solution for setting up a reliable test database each test without having to do any work. (I favour SQLite or similar.)

In Moodle 1.9 frankly I can only be bothered to do unit tests where really essential (e.g. I did a fix for a bug with the shorten_text function, which is a complete nightmare, so I needed a test to have some confidence in my fix not breaking everything). The rest of the time... meh. Yes, that definitely makes my code less reliable, but with our timescales and the difficulty of unit testing, I don't see any option.

OK so that was way too long; if you read this far, have a cookie. (No, I didn't bring any cookies! Supply your own. Sorry.) Still, that's what I currently think about unit tests, not being an expert but having done or attempted them for quite a while. Let's see if I change my mind after reading Tim's book! (Which by the way is 'XUnit Test Patterns')