Unit Tests vs. Integration Tests vs. Selenium Tests

The hardest question to answer, it seems, is what kind of test should we be writing? Within the industry today, we have a preconceived notion that integration tests are the best. When we have conversations with developers about what needs to be tested they like to say: test all of it. When we are hesitant that we may have broken something we turn to an integration or selenium test to verify it. This is a problem. We need to truly understand the cost and the confidence that we gain at each level of test. There are many mantras that say we need more unit tests. But the question is why? Also the question is why do we always think in terms of integrations and seleniums?

Confidence Over Time

Martin Fowler, presented a great post on the Test Pyramid. In the post, he speaks about he need to having a balanced testing suite, and the trade-offs for each type. So the question still remains: which do we use?

To answer this question, let's look at this from the perspective of the developer within two different contexts.

Time Paradox

The first context, is the traditional one: time. When we write code, often we found in a time crunch. We bid two days for writing this code, and now we are two days over. We are flustered and we haven't written any tests. Now we have a bunch of code that needs some tests and we need to get something tested quickly. As there are a lot of layers, we need to hit all of them. So, naturally we hand it off to QA or we ourselves write a selenium. We quickly roll out tests and we say: done. Shipped. We are good to go.

Now that we have those tests in place, we ship the code and bugs are found like crazy. We go back to make everything work, write another selenium or Integration to account for the different data passed in and we ship it. All the while the build is taking longer and we are still not testing what we think we are.

Confidence

The second context: Confidence. When we write code in this context, we ask ourselves: what Confidence am I still lacking? What am I worried about? If we are worried there might be a bug, we have ask the deeper question, where? We focus on the changes that we made. The risk. We focus on proving we didn't break something with our specific code. So the follow up question is: how can we do that with integrations or seleniums? We can't. We have to do that with units.

Inefficient Code Paths

See, the problem we are augmenting, is not that we are testing, we just are not testing what we think we are. It takes a comparatively long time to write seleniums and integrations. It takes a long time to run them and also a lot more setup to do it. They are costly. The most detrimental oversight is the inefficient code path. 

When we test logic paths from several layers away, we introduce inefficiencies and introduce assumptions. We need to ask ourselves: what am I testing? Are we testing the code path? Are testing the contract? Are we testing something else? Each question results in what to be tested. But hear he end all kicker: if we ever find a bug, the fix is always made in a function never in the system of code. So then we ask ourselves: why am I writing integrations and seleniums?

Reasons

There are reasons to write integrations and seleniums, but when we write tests with confidence in mind, we find these tests give it relatively small confidence in comparison to writing unit tests. So, these are the things we measure when we try to decide which to use.

So we make the decision, which is more important to our environment: time or confidence? I will say this: when confidence is your focus, time becomes a luxury.