October 17, 2019

10 Automated Testing Tools That Threat Stack Uses — and Why

Previously published on the Threat Stack Blog, September 19, 2019, co-author Laura Haiduck. 
All software development projects, whether they’re large or small, can benefit from well-planned and well-executed testing. It’s your way to ensure that the software you’re developing performs as expected and delivers value to the customer. More important — given the nature of our current cyber landscape — well-executed testing is your way to ensure that your software doesn’t ship with errors or vulnerabilities that could compromise its integrity. In a word, good testing lets you pass on performance value to customers — while also providing them with underlying security.

At Threat Stack, we have a complex, full stack development environment that requires a great deal of planning and decision making within the Test Engineering Team to make sure that all the functional areas of our platform are being tested effectively and efficiently. This includes decisions about what testing tools to use, since the right ones allow us to introduce automation as well as speed, repeatability, consistency, and accuracy of results.



In this post, we’re going to discuss ten of the automated testing tools we use —- with the goal of sharing some insights that, we hope, will help you in your testing activities.

Before looking at the tools, however, let’s create some context by examining key areas that our Test Engineers take into account when they’re testing our platform and its infrastructure.
  • Threat Stack agents are embedded within customers’ fleets of Amazon Web Services virtual machines in order to track and monitor them, whether they have Linux distributions — such as Ubuntu, Amazon Linux, Red Hat Enterprise Linux, CentOS, CoreOS — or whether they are of the Windows variety, such as Windows Server 2012 R2, 2016, or 2019.
  • Our customers might be using containers or Kubernetes that need to be monitored for signs of security incidents.
  • Each agent securely sends data to the Threat Stack Cloud Security Platform®, which is then analyzed and processed through one of our many microservices.
  • Event data is processed by Host, File, CloudTrail, Threat Intelligence, and custom rules to throw alerts, broken down by severity, to help our customers’ DevOps and Security teams analyze what is happening within their systems.
  • All the Event, Rule, and Alert data must be accessible not only through our browser-based application, but also through our Threat Stack APIs.
  • Our customers have the ability to take this data and export it into their own external storage to perform their own analysis.
  • Threat Stack Application Security Monitoring, a component of the Threat Stack Cloud Security Platform, runs as a microagent, watching application payloads, sending information to the Threat Stack platform, warning developers if it finds secure-coding mistakes. It also watches application payloads for malicious behavior such as cross-site scripting attacks.

As you can see, our software covers a lot of territory, and it has to do it fast, securely, accurately, and at scale. With that in mind, let’s take a look at some of the testing tools we use.
Testing Tools We Use at Threat Stack

Software Engineers in Test, embedded on their own feature teams, are not only responsible for testing new features and functions that are constantly being developed, but also for adding automation to make sure that the rest of the product still works as expected.

As a testament to the value our organization places on these people, each Test Engineer is also given the responsibility for recommending automation methods, tools, and languages that they deem appropriate. Below is a list of some of the tools we use to help us in our automated testing efforts. Please note that we use a more extensive toolset overall; our goal here is to discuss a selection of tools and to remind you to choose tools that work best for the specific projects you are working on.
Disclaimer: The resources mentioned in this post are not ranked in terms of perceived value. Nor do our comments imply an endorsement of any kind. Our intent is simply to provide you with information that we feel could be helpful as you add to your knowledge of testing tools.

1. Capybara


Website: https://teamcapybara.github.io/capybara/
Type: Free, Open source
Support: Ruby-Capybara Google Group

Latest version: 3.29.0 (September 2, 2019)

Capybara is the Ruby gem we use with Selenium in order to interact with our browser-based UI Automation tests when performing end-to-end user interface tests. Capybara contains built-in methods that emulate how a user can visit a page, fill_in a textbox, click_button, or select_dropdown.

The bane of a software tester’s existence is when it takes an absurd amount of time for web elements to load on a page, causing unexplained test failures. Each method provided by Capybara automatically has waits built in.

Capybara tests for user experience problems, operating our product much as our customers would, logging into our application, navigating through the site, viewing events triggered by our test systems, creating new rules that govern which events should trigger alerts, drilling down into alert information found on our dashboard, and updating various settings.

2. Gauge by ThoughtWorks


Website: http://gauge.org
Type: Free, Open source
Support: GetGauge Google Group, Gitter chat
Latest Version: 1.0.7 (August 29, 2019)

Gauge is a test automation framework created by ThoughtWorks Studio India, which we use to set up our UI and API Acceptance Tests. Although we pair Gauge with Ruby, it is also compatible with Java, C#, JavaScript, and Python. Gauge is much like Cucumber, the BDD framework, with minor differences:
  • Test plans are in specification files written in Markdown instead of Cucumber’s feature files, and this makes writing and maintaining tests easier.
  • Instead of Cucumber’s Given / Then / When format, test steps are listed as bullet points.

As a Behavior Driven Development (BDD) tool, Cucumber can become a bit wordy, the more complex the test is. With Gauge, each step is a clear, concise, easy-to-read bullet point. In addition, Gauge has built-in reporting available across multiple formats (XML, JSON, HTML). It can be used both on the command line and within an IDE such as Visual Studio Code. Finally, ample documentation is provided to cover a range of topics including installation, specification writing, example tutorials, and more.

3. Gatling Load Testing


Website: https://gatling.io/
Type: Both Enterprise and Open source versions
Support: Gatling Google Group
Latest Version: 3.2.1 (August 27, 2019)

Gatling provides continuous load testing here at Threat Stack. Written in Scala, it works either as a standalone tool, or can be used as a dependency to your product with Maven.

As a demonstration of its speed and power, within ten minutes we were able to generate three million requests in our test environment, and were able to determine that all of the requests returned results within under 800 milliseconds.

As BlazeMeter states in their blog, Eight Reasons You Should Use Gatling For Your Load Testing, “Gatling created detailed metrics dashboard that you can see after tests execution without having to add any additional plugins. The report is stored as an HTML file, which can be easily saved for some future analyses and metrics comparison. In addition to that, the report is interactive. which allows you to perform more detailed analyses and concentrate specific requests in addition to the overall picture.”

4. Test Kitchen by Chef.io


Website: https://kitchen.ci/
Type: Open source
Support: Chef.io Community
Latest Version: 2.2.0 (April 26, 2019)

Test Kitchen is the Chef.io product we use to automate spinning up and tearing down the different Amazon Web Services environments we use during our testing efforts. Test Kitchen is a test harness tool you can use to execute your configured code on one or more platforms in isolation. Many testing frameworks are supported by Kitchen out of the box, including Chef InSpec. Kitchen is used by all Chef-managed community cookbooks and is the integration testing tool of choice for cookbooks.

Learn.Chef.io has a well-thought-out series of training modules and demos, including an excellent module called Getting Started with Test Kitchen

5. ScalaTest


Website: http://www.scalatest.org
Type: Free, Open source
Support: ScalaTest Google Group
Latest Version: 3.0.8 (June 10, 2019)

Test Engineers at Threat Stack have started to partner with our Software Developers in order to write service tests for their Scala code using ScalaTest.

Our integration tests use the ScalaTest style trait, FeatureSpec, describing the test-driven development (TDD) style unit tests in a Given / When / Then format. As the ScalaTest Doc mentions, FeatureSpec is a “suite of tests in which each test represents one scenario of a feature. FeatureSpec is intended for writing tests that are ‘higher level’ than unit tests — for example, integration tests, functional tests, and acceptance tests.”

ScalaTest provides plenty of helpful documentation on their site, including a detailed quick start, installation procedures, and a user guide.

6. Chef InSpec


Website: https://kitchen.ci/docs/verifiers/inspec/
Type: Free, Open source
Support: Chef.io Community
Latest Version: (August 11, 2019)

At Threat Stack, we use Chef InSpec to create Ruby-based tests to verify that everything is working as expected once a test environment has been created on-the-fly. InSpec tests are meant to be human-readable. If you’re familiar with testing frameworks such as RSpec, InSpec code should be familiar. InSpec detects discrepancies and documents the differences in the form of a report. We can then go in and make the Chef changes needed to get a system to a desired state.

To get you started, the Learn Chef Rally has a useful segment that walks you through setting up a Docker and Docker Compose environment, downloading the sample environment, and exploring the environment using InSpec. 

7. Mocha


Website: https://mochajs.org/
Type: Free, Open source
Support: Mocha Gitter Group, Mocha Google Group
Latest Version: 3.2.1 (August 27, 2019)

Mocha is a JavaScript test framework that runs on Node.js and in a browser, enabling us to undertake asynchronous testing. Because Mocha tests run serially, it allows us more flexible and accurate reporting as well as the ability to map uncaught exceptions to the correct test cases.

Using Mocha in a Node.js environment, you can use the built-in assert module as your assertion library. At Threat Stack, we pair Mocha with Chai to create the test framework for the Threat Stack Application Security Monitoring functionality.

The Mocha site makes available an array of documentation on subjects covering everything from a detailed feature overview, installation, and getting started, to information on specific features and tasks. 

8. Cypress.io


Website: https://www.cypress.io/
Type:
  • Free, Open source test runner
  • Commercial Dashboard service, free for up to three users
Support: Gitter chat; Email support for paid users
Latest Version: 3.4.1 (July 29, 2019)

Cypress.io is built for JavaScript front-end developers who only need to test their work in Chrome. Cypress is executed in the same run loop as our application, meaning that all the functions, windows, web elements, timers, and services the Threat Stack application can access, our test code can access as well. We can also read and alter network traffic on the fly, which allows us to add some interesting negative test cases, such as testing what happens when a backend server is inaccessible.

Cypress.io offers well-organized documentation that provides a solid overview, getting started information, and core concepts that include information on writing and organizing tests. Looking to see what Cypress can do? TestAutomationU just released a new course, Introduction to Cypress.

9. Jest


Website: https://jestjs.io
Type: Free, Open source
Support: React Discord Chat, JestJs Google Group
Latest Version: 24.9 (August 2019)

Our full stack software developers have switched testing the Threat Stack Cloud Security Platform ReactJS-based frontend from Mocha / Chai to Jest, which has better built-in support for testing the ReactJS components we use. With this JavaScript testing framework, we are able to create better unit and integration tests. Both toolsets, ReactJs and Jest, are brought to you by Facebook.

As with most of the other tools in this post, the Jest website provides example-rich documentation covering how to get started, configuration, and a comprehensive range of use case scenarios.

10. Apache JMeter


Website: https://jmeter.apache.org/
Type: Free, Open source
Support: JMeter Google Group
Latest Version: 5.1.1 (March 13, 2019)

Apache JMeter is open source software designed to load test functional behavior and measure performance. According to their site, JMeter “may be used to test performance both on static and dynamic resources, Web dynamic applications. It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types.”

We use JMeter to analyze and measure the performance of our application and services. This allows us to test our application against heavy loads, with multiple and concurrent user traffic to see how we can improve our ability to scale.

JMeter makes available a useful array of documentation (including Getting Started, User Manual, and Best Practices) as well as a number of step-by-step tutorials.


As good as our Test Engineers are at Threat Stack, they wouldn’t get far without systematic processes backed up by powerful tools. As our platform evolves, our team continues to hone its expertise, continually upgrading its skills and knowledge, adopting and incorporating new tools, and developing new methods to ensure that they continue to add value to the Threat Stack SDLC. Hopefully this post has given you some insight into the ways our Test Team uses a range of automated testing tools to address the multiple functions within our Cloud Security Platform.


Happy Testing!

-T.J. Maher
Sr. QA Engineer, Software Engineer in Test
Meetup Organizer, Ministry of Testing - Boston

Twitter | YouTubeLinkedIn | Articles

17 comments:

Jessica said...

Nice blog buddy,

Thank you for sharing this amazing blog of automated testing tools, well for beginners your blog is really helpful i will also sharing it with my colleagues.

Here i am also sharing a blog on Automated Testing hope you will like it.

Anonymous said...

. A lot of software development teams are usually skeptical of the effectiveness of QA automation. When in fact, automation results and data unlock valuable insights that can potentially help securing better customer satisfaction and furnish the development team with a competitive edge. QA dashboards should be a one place stop for all the stakeholders, team members and management to gain an overall sense of the product development progress, it’s competitive advantage and goal fulfillment. Brickred System's UReport is a single solution for all the challenges in measuring QA business goals, providing a holistic framework to organize and measure the increasing number of complex QA activities while providing management with the QA metrics and KPIs they need to measure performance. For more information visit https://brickredsys.com/ureport/ and www.brickredsys.com/ureport

kirankumarpaita said...

software testing company in India
software testing company in Hyderabad
Thanks for sharing such a nice blog about Automated Testing Tools That Threat Stack Uses.
very informative and useful blog.
mainly helpful for software testers as beginners.
please keep sharing.

arshiya fouzia said...

This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more .
software testing course in chennai

Alia parker said...

Awesome work! I would like to say that typically a well-written article as we are seen here. This article is exceptionally valuable and I got so much data approximately program engineers. Much appreciated for sharing this article here. Web design and development service

seoaqsa said...

I want to say that this post is awesome, nice written and include almost all significant infos. I’d like to see more posts like this . best tools

racesitepro said...

Very informative blog article. Thanks Again. Keep writing. 경마사이트

totopickpro said...

Your style is very unique in comparison to other folks I ave read stuff from. Many thanks for posting when you have the opportunity, Guess I all just book mark this page. 토토

san987com said...

I think there are lots of more enjoyable instances ahead for individuals who take a look
at your blog post.

Stop by my web page - 우리카지노

Nila dharshan said...

Useful blog, keep sharing with us.

Important Features of JMeter
Scopes of JMeter

aaronnssd said...

Excellent post. I really enjoy reading and also appreciate your work. This concept is a good way to enhance knowledge. Keep sharing this kind of articles,Sell used tools in Pacific Beach Thank you.

Aarthi Ramesh said...

Useful blog, keep sharing with us.

Scope of JMeter
Why JMeter

Nancy said...

Before employing any SEO tools, it's critical to establish success measures to gauge your manual output, assess how your SEO tools improve your efficiency and decide if they are cost-effective. Philadelphia marketing agencies

klaus said...

These suggestions are so useful. It greatly aids me. It's a good idea to do business online. But we need to rank our business website in order to get more customers. And only Best Website Audit Service Agency is responsible for this. However, Google now checks who abides by its rules. therefore, we must adhere to their terms and improve our website.

Alia parker said...

Risk Management Software works by automating the process of risk assessment and mitigation. It uses data analysis and modeling techniques to identify potential risks, assess their likelihood of occurring, and prioritize their mitigation efforts.

johnson said...

Threat Stack has a sophisticated, complete stack development environment that necessitates extensive planning and decision making within the Test Engineering Team to ensure that all functional parts of our platform are properly and quickly tested. This includes decisions regarding which testing tools to use, because the appropriate ones enable us to integrate automation while also improving speed, repeatability, consistency, and accuracy of results.Thank you for sharing your knowledge! Keep up the good job! Continue to spread the word. Please take a peek at my website.
motorcycle accident near me

umeshtyagi said...

Ethical hacking includes an approved attempt to acquire unapproved admittance to a PC framework, application, or information. Completing an ethical hack includes copying the techniques and activities of pernicious aggressors. This training assists with distinguishing security weaknesses which can then be settled before a pernicious assailant has the chance to take advantage of them. The demand for cyber security professionals is high. Become a Certified Ethical Hacker to do the same, which a hacker does but legally. Enroll Now for Ethical Hacking Classes in Pune