February 8, 2016

Testing Beyond the UI: The Testing Pyramid

Problems with UI Tests


We've been using Selenium WebDriver with Java to write automated browser tests for our eCommerce application ever since the new Fitbit-Boston was formed two years ago.

We have tests for our web application emulating how online customers purchase goods, how customer service agents place orders, such as "Create a standard order containing a year long FitStar subscription and a Fitbit Blaze, with a billing address in the US and a shipping address in Canada, with overnight shipping, paying with a Discover Card".



We have browser tests that run every time code is merged into master, tests that run once per hour, and more tests that run just before a release candidate is deployed to production. We even have a nightly regression test suite of 320 some-odd tests that covers Firefox, Chrome, and Internet Explorer from IE9 to Microsoft Edge.

Even though our browser tests are pretty stable, executing on average with a 98% pass rate, there are some inefficiencies with the process:

  • Let's say we want to build a test that pressing the big shiny button actually captures a charge on one of our test credit cards, we need to go through the motions of placing an order first. If anything flaky happens with the web based user interface, the entire test fails, initially causing us to believe that maybe something was wrong with the capture charge functionality. 
  • Let's say we want to build a test that creates a refund. First, we need to place an order through the UI, then capture a charge. If anything flaky happens with those first two dependencies, the test fails. Likewise, we need to do some digging to figure out where the actual failure happened.   

If a certain test contained components that weren't specifically browser tests, it would be more efficient if we go one level deeper for those components, interacting with the same code the web browser is working from. We would work directly with the API (Application Programming Interface) and the application logic itself. Without a GUI (Graphical User Interface), these "headless" tests would not fail each time the properties of a web element changed.

Mike Cohn and the Testing Pyramid


Ever since the birth of Agile Software Development, two things were recognized with the short development cycles:

  • Automation should bear the burden of testing. 
  • Strategies with only UI automation will fail

Mike Cohn, author of "Succeeding with Agile: Software Development Using Scrum" (Nov. 2009) talks about in his book and later articles the concept of a "testing pyramid".



From The Forgotten Layer of the Test Automation Pyramid (12/17/2009):
"Even before the ascendancy of agile methodologies like Scrum, we knew we should automate our tests. But we didn’t. Automated tests were considered expensive to write and were often written months, or in some cases years, after a feature had been programmed. One reason teams found it difficult to write tests sooner was because they were automating at the wrong level. An effective test automation strategy calls for automating tests at three different levels, as shown in the figure below, which depicts the test automation pyramid [...]
"Automated user interface testing is placed at the top of the test automation pyramid because we want to do as little of it as possible. Suppose we wish to test a very simple calculator that allows a user to enter two integers, click either a multiply or divide button, and then see the result of that operation. To test this through the user interface, we would script a series of tests to drive the user interface, type the appropriate values into the fields, press the multiply or divide button, and then compare expected and actual values. Testing in this manner would certainly work but would be brittle, expensive, and time consuming. Additionally, testing an application this way is partially redundant—think about how many times a suite of tests like this will test the user interface. Each test case will invoke the code that connects the multiply or divide button to the code in the guts of the application that does the math. Each test case will also test the code that displays results. And so on. Testing through the user interface like this is expensive and should be minimized."

Martin Fowler and the Testing Pyramid

Martin Fowler agrees. Martin, according to his employee profile  at ThoughtWorks, "I'm an author, speaker… essentially a loud-mouthed pundit on the topic of software development. I've been working in the software industry since the mid-'80s where I got into the then-new world of object-oriented software. I got a call from ThoughtWorks in 1999 to consult on one of their projects and they rapidly became my favorite client. Within a year I was an employee".

From Martin's blog article, TestPyramid (5/1/2012):
"For much of my career test automation meant tests that drove an application through its user-interface. Such tools would often provide the facility to record an interaction with the application and then allow you to play back that interaction, checking that the application returned the same results. Such an approach works well initially. It's easy to record tests, and the tests can be recorded by people with no knowledge of programming. 
"But this kind of approach quickly runs into trouble, becoming an ice-cream cone. Testing through the UI like this is slow, increasing build times. Often it requires installed licences for the test automation software, which means it can only be done on particular machines. Usually these cannot easily be run in a "headless" mode, monitored by scripts to put in a proper deployment pipeline. 
"Most importantly such tests are very brittle. An enhancement to the system can easily end up breaking lots of such tests, which then have to be re-recorded".
An automated browser test suite has gone much farther than record-and-playback, as with Selenium IDE, but the problems remain:

When something fails, is it a problem with the functionality of the application itself? Or is it a problem with the browser? Do the browser tests need updating (again!) because a web element was modified slightly? Using more API tests would alleviate this problem.

An API Example

Take a look at the documentation for the online payment processor, Stripe at https://stripe.com/docs/api/ as an example as an Application Program Interface.

Want to create a new charge order? Get a list of previous charges? Process a return? No need to go through some type of user interface to interact with the system. You can use:

  • A programming language such as Java. Download their Stripe-Java module in their library to interact with their system.
  • Http calls to interact with their RESTful (representational state transfer) library. ( See Stripe's documentation with cURL.)

If we wanted to perform an API Test for Stripe, we could:

  • Create a new charge using certain parameters, and pass it to the Stripe API, returning a charge id.
  • Get the parameters of the charge created from the Stripe API using the charge id returned. 
  • Assert that the actual parameters match the expected parameters, failing the test if they do not match. 
No browser needs to be spun up for a Selenium WebDriver test, making it faster. There aren't any text fields or buttons or drop downs that needed to be pressed or selected. We don't need to worry about slow loading pages, or web elements that may or may not be there. We can focus just on the application itself. 

And how exactly do we apply this to our shopping cart tests?...

... I dunno. We are still working on that! :)


-T.J. Maher
 Sr. QA Engineer, Fitbit
 Boston, MA

// Automated tester for [ 11 ] month and counting!

Please note: 'Adventures in Automation' is a personal blog about automated testing. It is not an official blog of Fitbit.com








13 comments:

Dominick said...

The 2nd tale is fairly depressing as well as I practically really feel negative creating it, yet it should be placed out there. It is regarding a male that I recognized a while back that did a whole lot of taking a trip for the company pasco county recent arrests.

Roman Davis said...

It is will be a useful experiment for those people who are designers or web developers. I do not want to know more about these applications which are used during developing a website. Dissertation Help UK

Ariel Wilson said...

While I do think UI is one of the main aspects of digital products, I think it’s not the only thing that should be focused upon by the developers and testers. And while testing soft wares or applications, testers should look beyond UI. I haven’t read the complete post as I am busy writing a dissertation proposal that I have to submit tomorrow, but once I am done with that, I will definitely read the whole content – I really love the concept of testing beyond the UI.

SAVIOLA said...

Fantastic and unique article. I have enjoyed reading your points and have come to the conclusion that you are right about many of them., thanks for sharing. Also checkout fce abeokuta post utme registration form

Assignment Answers Online said...

We are a top rated assignment help online Online service here with experts specializing in a wide range of disciplines ensuring you get the assignments that score maximum grades.

assignmentauthors said...

Though we other sources that does this job well like this research work, it is vital I inform you that they are present.
theology assignment help
assignment help
computer science assignment help
capstone project help online
article writer
content writer for hire
content writing services online
freelance writer for hire
ghost writer for hire online
write my assignments for me

Essien said...

Your share is quite impressive. The information is also very helpful. All best wishes to you for your work. Thank you for sharing. Visit delta state college of health tech program/admission list

Rony James said...

It will be an interesting experiment for designers and web developers. I'm not interested in learning more about the applications that are used when creating a website https://www.lawessaywriters.co.uk/.

Anonymous said...

Amazing post. I suggest having a great time at 1 deposit casino Canada. Good luck

My Translation Services said...

Good post
Seeking professional translation services in Brighton? Look towards My Translation Services, the leading provider of linguistic expertise. With a meticulous eye for detail and a commitment to customer satisfaction, our skilled translators offer accurate and timely translations across various domains. From technical documents to marketing materials, we handle it all. Discover the difference with My Translation Services and unlock a world of global communication.

Anonymous said...

Discover how online assignment help by expert tutors can boost your knowledge and productivity in the fashion industry, particularly in designer fur coats womens. Share your experiences, get advice, and connect with like-minded professionals in our forum. Let's explore how online tutoring can enhance your skills and contribute to the success of your suede jacket business. Join the conversation now!

sopfiaa said...

Your breakdown of uncontested divorce costs in New York is invaluable. The clarity you provide on financial aspects eases the stress for those considering this step. Your blog is a reliable source, offering essential insights with empathy. Grateful for the guidance you bring to such a sensitive topic. Keep up the excellent work!
¿Cuánto Cuesta un Divorcio sin oposición en Nueva York?

Mike Rooney said...


This is just the information.

Dumb And Dumber In Suits