Why Most Unit Testing is Waste is an article written by Jim Coplien that was passed to me from a friend of mine. He wanted to know my thoughts on it.

I decided to title my article the same thing. And yes, it is basically clickbait... so, sorry. Whether you navigated here through rage or curiosity let it be clear that I believe unit testing is a very powerful and useful tool, but my views are my own. Now that we have that out of the way... I'd like to present my perspective on the article.

As a standing rule I'm always interested in opinions around testing methodologies and procedures, amongst a lot of other things. In this case, Jim, who I haven't spoken to before, is clearly an accomplished and intelligent person with a lot of real world experience to back up his claims. This is certainly not intended to say Jim is wrong, I'm sure he has his reasons and ultimately you will make up your mind either way based on your experience.

However, in a nutshell, the TLDR of my point of view is I think most of what Jim says is nonsense.

Sometimes It's Better to Ship With Bugs

Software engineering is an extremely broad and fast moving industry. It's easily broad enough to be split down in many sub-industries. Each of which may have spawned from the same engineering fundamentals but they follow very different disciplines and aim for very different goals.

On one end of this quality-to-effort-spectrum is mission-critical software where patches are not an option and a bug may be as catastrophic as a death. An example of this is the NASA Goddard Space Flight Center which was developed using the cleanroom engineering practice.

On the other end of that spectrum are some web applications. Some even used by millions of users everyday. This is the environment where patches can be released so regularly and with almost no effect to the users (other than fixing existing bugs, and hopefully not causing new ones). This is the opposite of mission-critical. In fact, this is a scenario where it actually makes sense to ship software with bugs.

Knowingly ship software with bugs!? In what universe does this make sense!? When the cost of fixing the bugs and the ability to ship it immediately and globally at no real expense. Even adding in the cost of potentially damaged reputation can all sum up to less than the cost it would take to fully test and QA that piece of software to the extent that your general user base effectively does for you by using it. This is a scenario that's not all that rare, and if the cost of the application comes down to it, it may just be crazy enough to make sense. Facebook is, or at least was, famous for this.

So why do I bring this up first? The goals of developing software are not always quality first. That's OK if you understand that going in. This is my disclaimer; I'm sure I come from a different part of that spectrum than Jim. That's neither good nor bad, but it is worth mentioning that all principles cannot be applied everywhere.

Most software would sit between these two points; where testing, of any kind, is beneficial to the application. I will come back to this at the end.

Newtonian Mechanics is Out. Quantum Mechanics is In.

"Reality is a Lovely Place... But I Wouldn't Want to Live There" - The Real World by Owl City

Newtonian mechanics provides us with a collection of simple and elegant formulae for calculating motion of large bodies. Quantum mechanics could be said to work the opposite. It's a universe of probability, infinity and down right craziness.

It's my belief that some software engineers find it difficult to reconcile, or outright deny, that their perfect newtonian world that they can feel safe inside of (information theory, computer science principles, architecture fundamentals) is continuously stomped on by the quantum world; which is more like the craziness of business practicality, ever changing, missing and contradicting requirements.

That doesn't mean that Newton was wrong, we still use his work to this day. My point is we need to get away from the text books that tell us how things should be and start adapting to the real challenges that are in front of use. The quantum world. Yes there is a whole set of maths there we don't understand, but we should embrace the learning and stop trying to disprove quantum reality with newtonian formulae because it's safe and familiar.

This is what my real issue with Jim's article is. As soon as phrases like "information theory", "Turing tape" and "J-Tag pin of a chip" come out to try and explain the issues with the real world I am always dubious.

I realise that it's unfair to cherrypick phrases out of context. Also, the irony is not lost on me that I'm also using an unrelated physics metaphor to carry my argument. But I'm going to stick with it.

You're a Genius. We Get It.

Much of your assumptions as based of the fact that code quality can or is improved by a utopian engineering team that has;

  • Unlimited time to plan out the correct solution.
  • Under strict and clear business requirements, so theirs no guessing.
  • The requirements do not change. Tests written that have been passing for a year can be thrown away (your words, not mine).
  • You have no crazy legacy or integration requirements to get in your way throughout your development lifecycle.

Let's bring it back to reality. You would be extremely lucky to get any one of those four. Even in that case it does not mean that your software will be superior quality to something developed under a completely different environment with different engineering skill sets and business pressures.

"Be humble about what your unit tests can achieve, unless you have an extrinsic requirements oracle for the unit under test. Unit tests are unlikely to test more than one trillionth of the functionality of any given method in a reasonable testing cycle. Get over it."

What an absurd and useless statement. That's like saying "I bought a new car and drove it home. However, until I drive it on every road on Earth I can't actually say it's able to drive." We write a unit test to validate a logical path:

int add(int a, int b) {
return a + b;
}

Yes, if the CPU is able to add two numbers together we can take for granted the fact that it can add any other two numbers together without testing every possible number combination.

"If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away. They are producing no information for you — or at least very little information. The value of the information they produce may not be worth the expense of maintaining and running the tests. This is the first set of tests to throw away — whether they are unit tests, integration tests, or system tests."

Deleting old tests is monumentally stupid. Throw all the information theory lingo you want at it. At the end of the day a feature developed may be working for over a year. Then there is a new business requirement to add extra functionality or update it; an extremely common scenario. All those tests you had, which you've now thrown away, will not give you feedback on if the the change you make now break existing logic.

"Tests should be designed with great care. Business people, rather than programmers, should design most functional tests."

This is another fantasy; the part about business people writing tests. I have even tried to subscribe to this myself, more than once. Back when I was working in a team working with Agile, user stories, BDD, the whole kit and kaboodle. Business people do not care [about writing functional tests], and they certainly don't have the time to sit with engineers and hash out system tests. In most cases you would be lucky to get high level system system functionality out of them because they don't know either.

"Maybe the requirements are so bad that developers wouldn’t know what to test if they had to, so they make their best guess. Software engineering research has shown that the most cost- effective places to remove bugs are during the transition from analysis to design, in design itself, and in the disciplines of coding. It's much easier to avoid putting bugs in than to take them out."

You hit the nail on the head. This is the real world for the vast majority of engineers. It doesn't have to be a bad thing, our experience is what matters here. Being able to adapt to that environment and continuously improve our surrounding without reverting back tot he text book and dictating how things should work.

Enough Negativity

I want to have a beer with Jim. Not to argue over this stuff, but to better understand where he is coming from and why his vast experience has brought him to those conclusions. There is no doubt in my mind that there is much to learn. As I mentioned at the start this wasn't meant to be a roast.

I had to read his article several times, and sleep on it a few more times before I started to grasp his message between the lines. I believe that message to be along the lines of "don't blindly follow process for process sake. Use your experience and keep and open mind about what's important to your products needs." That's something I definitely agree with can get behind - even if it doesn't provide the black or white solution we are all seeking to find.

P.S. It will definitely take more than one beer.