February 8, 2016

Testing Beyond the UI: The Testing Pyramid

Problems with UI Tests


We've been using Selenium WebDriver with Java to write automated browser tests for our eCommerce application ever since the new Fitbit-Boston was formed two years ago.

We have tests for our web application emulating how online customers purchase goods, how customer service agents place orders, such as "Create a standard order containing a year long FitStar subscription and a Fitbit Blaze, with a billing address in the US and a shipping address in Canada, with overnight shipping, paying with a Discover Card".



We have browser tests that run every time code is merged into master, tests that run once per hour, and more tests that run just before a release candidate is deployed to production. We even have a nightly regression test suite of 320 some-odd tests that covers Firefox, Chrome, and Internet Explorer from IE9 to Microsoft Edge.

Even though our browser tests are pretty stable, executing on average with a 98% pass rate, there are some inefficiencies with the process:

  • Let's say we want to build a test that pressing the big shiny button actually captures a charge on one of our test credit cards, we need to go through the motions of placing an order first. If anything flaky happens with the web based user interface, the entire test fails, initially causing us to believe that maybe something was wrong with the capture charge functionality. 
  • Let's say we want to build a test that creates a refund. First, we need to place an order through the UI, then capture a charge. If anything flaky happens with those first two dependencies, the test fails. Likewise, we need to do some digging to figure out where the actual failure happened.   

If a certain test contained components that weren't specifically browser tests, it would be more efficient if we go one level deeper for those components, interacting with the same code the web browser is working from. We would work directly with the API (Application Programming Interface) and the application logic itself. Without a GUI (Graphical User Interface), these "headless" tests would not fail each time the properties of a web element changed.

Mike Cohn and the Testing Pyramid


Ever since the birth of Agile Software Development, two things were recognized with the short development cycles:

  • Automation should bear the burden of testing. 
  • Strategies with only UI automation will fail

Mike Cohn, author of "Succeeding with Agile: Software Development Using Scrum" (Nov. 2009) talks about in his book and later articles the concept of a "testing pyramid".



From The Forgotten Layer of the Test Automation Pyramid (12/17/2009):
"Even before the ascendancy of agile methodologies like Scrum, we knew we should automate our tests. But we didn’t. Automated tests were considered expensive to write and were often written months, or in some cases years, after a feature had been programmed. One reason teams found it difficult to write tests sooner was because they were automating at the wrong level. An effective test automation strategy calls for automating tests at three different levels, as shown in the figure below, which depicts the test automation pyramid [...]
"Automated user interface testing is placed at the top of the test automation pyramid because we want to do as little of it as possible. Suppose we wish to test a very simple calculator that allows a user to enter two integers, click either a multiply or divide button, and then see the result of that operation. To test this through the user interface, we would script a series of tests to drive the user interface, type the appropriate values into the fields, press the multiply or divide button, and then compare expected and actual values. Testing in this manner would certainly work but would be brittle, expensive, and time consuming. Additionally, testing an application this way is partially redundant—think about how many times a suite of tests like this will test the user interface. Each test case will invoke the code that connects the multiply or divide button to the code in the guts of the application that does the math. Each test case will also test the code that displays results. And so on. Testing through the user interface like this is expensive and should be minimized."

Martin Fowler and the Testing Pyramid

Martin Fowler agrees. Martin, according to his employee profile  at ThoughtWorks, "I'm an author, speaker… essentially a loud-mouthed pundit on the topic of software development. I've been working in the software industry since the mid-'80s where I got into the then-new world of object-oriented software. I got a call from ThoughtWorks in 1999 to consult on one of their projects and they rapidly became my favorite client. Within a year I was an employee".

From Martin's blog article, TestPyramid (5/1/2012):
"For much of my career test automation meant tests that drove an application through its user-interface. Such tools would often provide the facility to record an interaction with the application and then allow you to play back that interaction, checking that the application returned the same results. Such an approach works well initially. It's easy to record tests, and the tests can be recorded by people with no knowledge of programming. 
"But this kind of approach quickly runs into trouble, becoming an ice-cream cone. Testing through the UI like this is slow, increasing build times. Often it requires installed licences for the test automation software, which means it can only be done on particular machines. Usually these cannot easily be run in a "headless" mode, monitored by scripts to put in a proper deployment pipeline. 
"Most importantly such tests are very brittle. An enhancement to the system can easily end up breaking lots of such tests, which then have to be re-recorded".
An automated browser test suite has gone much farther than record-and-playback, as with Selenium IDE, but the problems remain:

When something fails, is it a problem with the functionality of the application itself? Or is it a problem with the browser? Do the browser tests need updating (again!) because a web element was modified slightly? Using more API tests would alleviate this problem.

An API Example

Take a look at the documentation for the online payment processor, Stripe at https://stripe.com/docs/api/ as an example as an Application Program Interface.

Want to create a new charge order? Get a list of previous charges? Process a return? No need to go through some type of user interface to interact with the system. You can use:

  • A programming language such as Java. Download their Stripe-Java module in their library to interact with their system.
  • Http calls to interact with their RESTful (representational state transfer) library. ( See Stripe's documentation with cURL.)

If we wanted to perform an API Test for Stripe, we could:

  • Create a new charge using certain parameters, and pass it to the Stripe API, returning a charge id.
  • Get the parameters of the charge created from the Stripe API using the charge id returned. 
  • Assert that the actual parameters match the expected parameters, failing the test if they do not match. 
No browser needs to be spun up for a Selenium WebDriver test, making it faster. There aren't any text fields or buttons or drop downs that needed to be pressed or selected. We don't need to worry about slow loading pages, or web elements that may or may not be there. We can focus just on the application itself. 

And how exactly do we apply this to our shopping cart tests?...

... I dunno. We are still working on that! :)


-T.J. Maher
 Sr. QA Engineer, Fitbit
 Boston, MA

// Automated tester for [ 11 ] month and counting!

Please note: 'Adventures in Automation' is a personal blog about automated testing. It is not an official blog of Fitbit.com








9 comments:

ram said...

Worth reading! Our experts also have given detailed inputs about these trainings & courses! Presenting here for your reference. Do checkout
aws training in chennai & enjoy learning more about it.

Dominick said...

The 2nd tale is fairly depressing as well as I practically really feel negative creating it, yet it should be placed out there. It is regarding a male that I recognized a while back that did a whole lot of taking a trip for the company pasco county recent arrests.

Qasim Khan said...

I found your website perfect for my needs. I always like your article because you have provided every time informative post.Pay for Essay Writing

Devi said...

Want to set your career towards Big Data? Then Infycle is with you to make this into your life. Infycle Technologies gives the combined and best Big Data Hadoop training in Chennai, along with the 100% hands-on training guided by professional teachers in the field. In addition to this, the mock interviews for the placement will be guided to the candidates, so that, they can face the interviews with full confidence. Once after the mock interview, the candidates will be placed in the top MNC's with a great salary package. To get it all, call 7502633633 and make this happen for your happy life.Big Data Hadoop Training in Chennai | Infycle Technologies

Roman Davis said...

It is will be a useful experiment for those people who are designers or web developers. I do not want to know more about these applications which are used during developing a website. Dissertation Help UK

Ariel Wilson said...

While I do think UI is one of the main aspects of digital products, I think it’s not the only thing that should be focused upon by the developers and testers. And while testing soft wares or applications, testers should look beyond UI. I haven’t read the complete post as I am busy writing a dissertation proposal that I have to submit tomorrow, but once I am done with that, I will definitely read the whole content – I really love the concept of testing beyond the UI.

MARKETING ASSIGNMENT HELP said...

Do you also want Assignment help? We lend expert writers for assisting you with the projects. Our qualified writers will assist you with everything. You will get 100% original content within the deadline. Hire our assignment helper online and finish your project works.

Unknown said...

Data Science Certification in Chennai for skyrocketing your career with Infycle Technologies, the best Software Training & Placement institutes in and around Chennai. In addition to the Certification, Infycle also gives the best placement training for personality tests, interview preparation, and mock interviews for leveling up the candidate's grades to a professional level.

Unknown said...

Infycle Technologies, the top software training institute and placement center in Chennai offers the Best Digital Marketing course in Chennai for freshers, students, and tech professionals at the best offers. In addition to Digital Marketing, other in-demand courses such as DevOps, Data Science, Python, Selenium, Big Data, Java, Power BI, Oracle will also be trained with 100% practical classes. After the completion of training, the trainees will be sent for placement interviews in the top MNC's. Call 7504633633 to get more info and a free demo.