Test Your Code

Have You Ever Experienced This?

Part of my work is fixing bugs. Some bugs are really hard to track down and it’s easy to see how those errors have been overlooked during implementation.

However there is a small percentage of bugs that are caused by code, that just plain obviously cannot possibly work under any circumstances. Regardless of the input, the code just always has to fail – every single time. Quite frankly, I am a little annoyed, when I run into one of these.

To be clear: I am not annoyed by programming mistakes. They happen to me more often than I am willing to admit. I am annoyed by the fact that someone feels it’s acceptable to commit and ship code, that has not been executed even once.

A Little Mathematical Analysis

Let’s take a mathematical approach to this. Given some assumptions, how high is the probability to create a bug-free release when skipping testing?

If I assume, I manage to get a method correct without testing 19 out of 20 times and I skip testing for 10 methods. (The math assumes independence – which is not entirely correct, but good enough for our purposes).

Probability a given method is correct: P_m = 0.95

Number of methods for which testing is skipped: n = 10

Probability release is correct: P_r = P_mⁿ = 0.95^10 = 0.60

This gives us a 60% probability we get a good release. Or in other words a 40% probability to generate some production problem and require a patch.

The numbers used here are probably too good. Probably I don’t get 19 out of 20 methods right on the first go. Also a typical release has way more than 10 methods. If we lower P_m or if we increase n, things get bad very quickly. Consider e.g. the following examples:

1. For P_m=0.90 and n=10 there is only 0.35 chance the release will be bug free.
2. For P_m=0.95 and n=20 there is only a 0.36 chance the release will be bug free.
3. For P_m=0.90 and n=20 there is only a 0.12 chance the release will be bug free.

A table with some sample values is included below for your reference.

The Incremental Cost of Fixing a Bug

The steps to fix a bug depend greatly on how early it is identified. E.g.:

1. If the bug is identified by the engineer while he is writing the method and testing it in isolation, he can fix it very quickly. The method is still open in the IDE and it is only a couple of lines long. He just noticed what went wrong and it takes him a couple of minutes at most to fix it.
2. If the bug is found by the engineer when testing a whole feature end to end, things take a little longer. Maybe he has to do additional testing to track the bug down, check the stack trace, identify the faulty method and work on code that he might have written a few days ago. Still the problem is fixed typically in a couple of hours at most.
3. If the bug is found by a different engineer, while he is working on a different but somewhat related feature, he has to track the bug down, find the faulty method, analyze the code and fix it. Needless to mention if this happens a few times it’s not exactly helping the team spirit.
4. If the bug is found by the testing team before shipping a release, the same thoughts as with point 2. and 3. apply. In addition a ticket has to be raised in the bug tracking tool and it has to be prioritized, also the release might get delayed last second.
5. If the bug is found by the customer after the release has been shipped, things can get really messy. On top of all the work of the previous point, a new release has to be built, shipped and deployed. Unfortunately for big corporations deploying a release can be a complicated process, that takes weeks or even months of elapse time. Also the customer has to deal with faulty software. This might result in reputational damage for both the software vendor and the customer. Worse, it might result in actual, immediate financial losses for the customer, if e.g. the bug affects critical processes or if decisions are based on erroneous data.

In summary a bug is really cheap to fix if it is found early. And it’s really expensive to fix if it is found late. As testing early is really simple to do and takes little time it is obvious to see why testing early is a smart move.

Just Start Testing

Sometimes teams don’t take action towards testing because they are intimidated by the sheer number of available options and tools or the associated license fees.

Therefore I suggest to just start with the simplest, most straight forward approaches to testing. When you are working on code just test it. Keep the overhead as low as possible.

• If you are working on a web application, just run the application on a local web server as soon as you’ve implemented a usable increment and test it in your browser.
• If you are working on a desktop application, just install it (locally or on a VM) as soon as you’ve implemented a usable increment and test it.
• If you are working on a mobile application, just install it on a phone as soon as you’ve implemented a usable increment and test it. Alternatively you can do the testing on a simulator.
• If you are working on a Web service, just call the web method using a tool like SoapUI. SoapUI is a great tool to call and test Web Services. It’s powerful and it’s very easy to use. If you’ve never used it before, you will be ready to run your first tests within hours. To the best of my knowledge the open source version is available for free. And yes, it supports REST as well.
• If you are working on a command line tool, just call it from the command line as soon as you’ve implemented a usable increment and test it.
• If you are working on a library, just write a simple command line tool that calls the library. So you can call and test the public methods of your library and get instant feedback.

All these approaches are really easy to implement. They don’t require any management decisions, expensive tools or extensive discussions. They can be implemented by developers immediately and improve quality of the software significantly.

To be clear, I think there is value in approaches like e.g. automated GUT testing . However I think these techniques should not be the first step towards testing, as they require a bigger initial investment.

Conclusion

• Bugs found early are cheap to fix. Bugs found late are expensive to fix.
• Basic testing can be done very easily and very quickly. No expensive tools are required. No complicated systems or techniques have to be learned.
• Start with simple testing approaches immediately.