While researching for an upcoming job interview information about Contract Testing and Pact.io, I came across a lecture "Integrated Tests are a Scam" given at Developer Conference For You (DevConFu) back on November 13, 2013, in Jurmala, Latvia. It's amazing what historical records one can find on the internet!
I found a blurb on Pact.io / History that when Pact.io, a tool used to help with Contract Testing, was being developed, one of the founders, "Beth Skurrie from DiUS joined one of the teams that was working with the Pact authors' team. She had recently seen a talk by J. B. Rainsberger entitled 'Integration tests are a scam', which promoted the concept of 'collaboration' and 'contract' tests, so she was immediately interested when she was introduced to Pact". This blurb intrigued me, so, of course, I had to find a copy of this talk.
J. B. (Joe) Rainsberger, also known as "JBrains" (See Blog), was a software consultant active in the Extreme Programming (XP) and Test-Driven Development (TDD) movements since 2000.
Below are my research notes on Joe Rainsberger's lecture:
"Integrated Tests are a Scam: A self-replicating virus that invades your progress. It threatens to destroy your codebase, to destroy your sanity, to destroy your life".
An Integrated Test, Joe says, is where the pass or fail behavior could be because of many different things that are happening at the same time. Nothing is isolated in the test. You can't just automatically see where the error can be found, and what exactly is causing it. And is it really testing, or is it more checking? Referencing "Checking vs Testing" by Michael Bolton ("No, not the singer. Not the character from Office Space. The third one").
Why does Joe call it a scam? Unit tests might pass, but bugs may only be found by testing the software product as a whole.
If a bug is found with an integrated test, we may realize we don't have enough coverage, so we write more tests. Test Driven Development's purpose is to examine the quality of the design by writing isolated tests. Integrated tests test big parts of the system and don't put pressure on the design, so the more integrated tests we have, the less design feedback we get. The design might get sloppy, according to Joe. Certain parts depend more strongly on other parts. Change one item and another item magically breaks. It is easier to make mistakes, so we need more integration tests. How many? It depends on how many components are talking to one another -- and, as Joe describes in the talk, there is no reliable way of knowing whether you have written enough.
Example: With two components: If Object A has 10 tests, and Object B has 8 tests, in order to test how A talks to B, you would need 80 integrated tests for the combination. Instead, we look at an interface and describe "what can I do" without describing "how do I do it"? The set of behaviors of these functions is the Contract, describing what is true about the interface. You can describe how Object A and Object B should work together without actually putting them together. Whenever you have a function that can return a collection of data:
"There are five types of test that we should write:
- Zero.
- One.
- Many.
- Lots.
- Oops."
What happens if you ask for a data collection and it returns zero results? One result? Many results? Lots and lots of results, or an error when returning results?
Joe also introduces in his lecture Mock Objects, something he prefers to call "test doubles", like a movie's stunt doubles. They have expectations, such as telling a request what is expected to happen for a test. They have answers which have stubs, hard-coding a specific answer convenient for the test. You can have stubs that return empty data sets.
State based tests are: You have an item, put the item in a shopping cart -- is the item there?
Paraphrasing, a collaboration test is designed to answer: do the objects talk to their neighbors correctly? And what happens if the thing you are talking to has different behavior than expected, such as returning "23" instead of "22"? That is a contract problem.
Contract Tests: Does my server implement my interface "correctly", and does it respect the contract?
If you have a test that asks the model to find all customers registered in 1973, there needs to be a test on the database implementation of the model: what happens when someone makes that request.
If Object A says, "I have a question!", Object B needs to say, "Here's how I answer that question". Expectations and Actions need to match. Answers and Assertions need to match.
After this exercise is complete, most of the tests that have been written are Unit Tests, with only a thin layer of integrated tests at the boundaries.
The tests are mainly Isolated Object tests:
- Simple state based tests: Create an Object, Call a Function, Assert Equals.
- Collaboration + Contract Tests: Set expectations and make sure those expectations in the data layer above match actions in the layer below. Stubs in the layer above match the assertions in the layer below.
Integration tests are slow. They need to talk to databases. They are brittle -- one problem not related to a test, such as a service being down, could make 37 tests fail.
The tests duplicate a bunch of stuff. "They are slow, brittle, and you need to write a ton of them. We have a combinatorial explosion problem."
Unit tests are tiny, easy to understand, and very quick to run.
"If you are finding that collaboration tests are hard to write, your design is flawed.
"It means your interactions are too complicated.
"That means you design more carefully, and when you design more carefully, then you make fewer mistakes.
"And when you make fewer mistakes, then you have more energy in which to write more unit tests [...]
"Now, we are in a Virtuous Cycle. And that, my friends, is how you beat the scam of integrated tests. You don't need them."
... I can see why someone who was into Contract Testing was inspired by this lecture!
Happy Testing!
-T.J. Maher
Software Engineer in Test
BlueSky |
YouTube |
LinkedIn |
Articles
No comments:
Post a Comment