How to add Static Application Security Testing (SAST) with SonarQube Cloud Free Tier to a GitLab CI/ CD pipeline

ESLint catches code quality issues. Typecheckers ensures variables and functions use the correct data types. Prettier automatically formats code for consistent style. But how can we start incorporating security testing into this GitLab CI/CD pipeline we have built? 

• GitLab repo: https://gitlab.com/tjmaher/bun-create-playwright

For this blog post we will be looking at one way to do Static Application Security Testing (SAST), adding SonarQube to our project.

What is Static Application Security Testing?

Static Application Security Testing (SAST) is a method of security testing that scans an application's source code, bytecode, and binaries for any security flaws. Because it analyzes the code without actually running the application -- meaning the code is "static" -- developers can use it early in the coding phase to catch and fix bugs immediately.

Although source code scanners have been around a while in the computer industry,  the term "Static Application Security Testing (SAST)" originated as a buzzword by Gartner, Inc. Gartner is known for their "Magic Quadrant" a visualization technique they use to map out industry leaders on a four part grid "Completeness of Vision" and "Ability to Execute" showing who are the market leaders, who are the challengers, who are the visionaries, and who are the niche players, as they put it.

Joseph Feiman and Neil Macdonald published in 2009 their paper Magic Quadrant for Static Application Security Testing (See PDF). 

Refactoring The-Internet: Deploying the web app to Render

We are well on our way to place a copy of my favorite test site, The-Internet, onto its new home on Render.com, a place where developers can deploy web apps using their free tier, much like Heroku in its heyday. Sure, web apps that live on the free tier of Render takes a minute or so to spin up, but, hey, you get what you pay for!

With our last blog post, we: 

• Examined using Google AI the Gemfile of the original The-Internet, a Ruby project at https://github.com/saucelabs/the-internet, hosting on the failing site, https://the-internet.herokuapp.com/
• Created a new copy of the web app onto my GitHub site, forking the original repo
• Updated the Ruby architecture of the web app per Google AI's findings. 
• Tested the web app could be deployed locally and pushed the changes to my GitHub site: https://github.com/tjmaher/the-internet
• Almost have the site at its new home at https://the-internet-8uaj.onrender.com/

With this blog post, we will create a new account on Render, a new project, and set up a new home for The-Internet.  

Publishing Playwright HTML reports on GitLab Pages

We've added a lot to our demo project, bun--create-playwright. 

We've explored setting up a Playwright framework using bun, a new package manager bundled into Claude Code. We've added typechecking, and formatting and linting. We've set up a CI/ CD pipeline for our tests using GitLab, one with three stages: 

• Quality: We check that the formatting, linting and typechecking is correct for any changes we attempt to push to the code base.
• Test: We run the smoke and then the regression tests to make sure that everything still works. 
• Report: We bundle an archive of the Playwright generated HTML report, with screenshots and videos if things fail.

For this post, we will add a new stage to this GitLab pipeline:

•  Deploy: Where we will deploy the HTML report to GitLab Pages, so we can view it online. 

Examining the artifacts job of a Playwright GitLab CI / CD pipeline

You may have noticed that in the GitLab CI / CD Pipeline for our Playwright project, in the .gitlab-ci.yml file, after our scripts has run, in the Test stage, there is a subsection called "artifacts" with certain paths to something called "playwright-report", "test-results", and "reports". 

The Playwright -> Artifacts -> Reports stage

playwright:

  extends: .bun_playwright

  stage: test

  timeout: 30 minutes

...

...

  artifacts:

    when: always

    paths:

      - playwright-report/

      - test-results/

      - reports/

    reports:

      junit: reports/junit/results.xml

    expire_in: 30 days

https://gitlab.com/tjmaher/bun-create-playwright/-/blob/main/.gitlab-ci.yml

Every time Playwright tests run, it generates proof of the test execution: screenshots capturing the state of the UI when a test failed, video recordings of an entire browser session, HTML and JUnit XML test reports, trace reports showing action logs, network requests. These artifacts, when the tests run locally, are placed by the Playwright Test runner in generated folders, playwright-report and test-results, along with a folder called reports/junit. 

With this post, we will be examining how this artifacts stage of the GitLab CI/ CD pipeline produces downloadable artifacts we can examine.

Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report

So far, we have reviewed how to review code with ES Lint + Prettier and Typecheck, how to set up and run smoke tests, how to run all Playwright + TypeScript tests, and reviewed the HTML report of results.

In this post, we are going to set up a three stage GitLab CI/ CD pipeline that will run against every merge request: 

•  Quality (lint, typecheck, prettier)  --> Test ( smoke + regression ) --> Report ( Downloadable )

I haven't used GitLab since when I worked at ThreatStack back in 2020. ( See my blog entry Getting to Know GitLab and How They Test the UI )

GitLab reads .gitlab-ci.yml from the repository root and turns it into a pipeline, a set of jobs, grouped into stages, running whenever you push a change to a repository. 

Stages run one after another, and jobs within a stage run in parallel.  If one stage fails, the stages after it are skipped. 

This begs the question: What is GitLab? What is CI/ CD? Or a pipeline? Or a merge request? 

How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json

Earlier, we went over how we could create scripts in the package.json file of our Playwright framework to add typechecking, linting, and formatting your code with prettier. 

• Want to see the entire bun-create-playwright project? It is on tinyurl.com/bun-create-playwright!

With this post, we will explore how the built-in test runner for Playwright Test can run headed tests, debug versions of tests, and smoke tests.

... and shortcuts for all of these can be set up in the scripts in your package.json! If you are using "bun" as a package manager, as we are in bun-create-playwright, just type out "bun run", a space and then the shortcut such as: bun run test

package.json

"scripts": {

    "test": "playwright test",

    "test:headed": "playwright test --headed",

    "test:trace": "playwright test --trace on",

    "test:chromium": "playwright test --project=chromium",

    "test:firefox": "playwright test --project=firefox",

    "test:webkit": "playwright test --project=webkit",

    "test:smoke": "playwright test --project=chromium --grep '@smoke'",

    "test:flaky": "playwright test --project=chromium --repeat-each=20",

    "test:ui": "playwright test --ui",

    "test:debug": "playwright test --debug",

    "test:failed": "playwright test --last-failed",

    "test:login": "playwright test tests/login.spec.ts",

    "test:secure-area": "playwright test tests/secure-area.spec.ts",

    "report:list": "playwright test --reporter=list",

    "report:line": "playwright test --reporter=line",

    "report:dot": "playwright test --reporter=dot",

    "report:blob": "playwright test --reporter=blob",

    "report": "playwright show-report",

    "codegen": "playwright codegen",

    "lint": "eslint .",

    "lint:ci": "eslint . --max-warnings 0",

    "lint:fix": "eslint . --fix",

    "format": "prettier --write .",

    "format:check": "prettier --check .",

    "format:debug": "prettier --check . --log-level debug",

    "format:diff": "prettier --list-different",

    "typecheck": "tsc --noEmit"

  },

bun-create-playwright / package.json

Do you need to really set up shortcuts like these? Certainly not! But it is easier than typing out: bunx playwright test --project=chromium --grep '@smoke'.

Feel free to name these commands anything you want! 

Introducing bun, a new package manager and JavaScript runtime environment

New job? New toolset to explore! In this post I'll be investigating a new JavaScript package manager and runtime environment called bun.sh. 

You may have noticed on Playwright.Dev's Getting Started / Installation section there are three different ways to install a ui or api test automation framework with the latest version of Playwright:

• npm init playwright@latest
• yarn create playwright
• pnpm create playwright

For this blog post I'll be investigating a fourth way, the new toolset they just started using at work, bun.sh.

• bun create playwright

What is npm, yarn, pnpm, and bun? These are called package managers.

What is a package? Developing a project, you don't have to figure out how to code everything yourself. You can include in your projects outside libraries, or "packages" of code from from the public JavaScript registry npmjs.com. 

The problem is that these packages then need to be downloaded, installed, managed, and updated whenever they are updated. JavaScript projects can use a package manager, such as bun, to help this process. 

When putting together a Playwright automation framework based around JavaScript and TypeScript, Playwright uses the Node.js JavaScript runtime environment, to run its test runner, execute automation scripts, and manage its browser instances.(See Nodejs.org / About )

There is a division of labor: Node.js runs these internal functions like running tests. NPM, Yarn, PNPM, and now Bun handle the packages. 

Why are there so many package managers? Each has its own specialization. The original NPM is bundled already with Node.js. Yarn is a Facebook toolset that was created when NPM was found to work too slow. PNPM was created to be smaller and therefore faster. And bun was built for quick startup and installation speed, good especially if you have a CI/ CD pipeline that has a LOT of packages to install each time the tests run.

Practicing Playwright: Visual Testing With Playwright

If you want to do some basic visual checking to see if there has been any deviation from your baseline of images, you can use Playwright. It's built in! Playwright can take a snapshot of a web element, a visible viewport, or a full page, and save it in your Git repository as a baseline, failing the test if the image, page, or viewport does not match up. 

Caution: From what I have been reading, this can quickly cause your code repository to balloon in size, since Chrome, Firefox, and WebKit would each store its own golden screenshot in your repo. Also, images on Mac, Windows, and Linux all appear different pixel-by-pixel. If using a CI platform, it might be best to run visual tests only on a standard Playwright Docker image, to generate and compare snapshots. According to TestQuality, "Once a suite passes 50–100 visual tests, teams need a layer that tracks run history, surfaces flaky-test patterns across cycles, and routes confirmed defects into the team's tracker — none of which lives inside the test runner itself"... I wonder if you can store images in an Amazon S3 bucket and hook that up as a virtual drive? ... no matter. That will be a blog post for another time... 

Right now, we will be walking through Butch Mayhew's code he wrote for his LinkedIn Learning course, Learning Playwright, found on his companion GitHub site. 

While the test is in a certain state you can a screenshot of the page or certain elements of the page and save them as a snapshot. The snapshots an be used as a baseline images to compare your current site against. This baseline can be periodically updated as the site evolves. 

How does this happen? With Playwright's .toHaveScreenshot( ) to take a screenshot and the mask method that you want to leave out of the comparisons between the expected and actual screenshots. 

• toHaveScreenshot(name): Playwright.dev / Page Assertions

The History of Contract Testing with Pact.io

Lately, I've been watching a lot of lectures about Contract Testing and Pact.io, trying to prepare for an upcoming job interview. When diving into a new toolset I can never simply jump into the code. I need to know: Why was this toolset created? What problem did it solve? How was this tool created? How did this toolset evolve?

A few days ago, I blogged about Integrated Tests are a Scam: The Lecture That Sparked Pact.io talking about J. B. Rainsberger's 2013 lecture. Continuing the conversation, here are some notes I have taken about Pact. 

10 Automated Testing Tools That Threat Stack Uses — and Why

Previously published on the Threat Stack Blog, September 19, 2019, co-author Laura Haiduck. 

All software development projects, whether they’re large or small, can benefit from well-planned and well-executed testing. It’s your way to ensure that the software you’re developing performs as expected and delivers value to the customer. More important — given the nature of our current cyber landscape — well-executed testing is your way to ensure that your software doesn’t ship with errors or vulnerabilities that could compromise its integrity. In a word, good testing lets you pass on performance value to customers — while also providing them with underlying security.

At Threat Stack, we have a complex, full stack development environment that requires a great deal of planning and decision making within the Test Engineering Team to make sure that all the functional areas of our platform are being tested effectively and efficiently. This includes decisions about what testing tools to use, since the right ones allow us to introduce automation as well as speed, repeatability, consistency, and accuracy of results.

Notes: Selenium 4 With Simon Stewart

Simon Stewart, the Selenium product lead, the creator of WebDriver gave a brief preview on what is coming up with Selenium 4.

Selenium, Simon says, is a family: Selenium WebDriver, Selenium Grid, and now Selenium IDE.


https://youtu.be/6iHdvOYdJk8

My Threat Stack interview: What are the current trends in automation?

Threat Stack, where I work, has started a campaign to showcase on its company blog not just its developers and security analysts, but also its software testers.

I was interviewed for Threat Stack's article, Current Trends in Automated Software Testing: 10 Testing Pros Share Insights Into the Most Interesting Current Trends in Automated Software Testing.
My TLDR version:

When @ThreatStack inquired about the biggest trend in automation, I responded, "Free Education for all!"
* @TechGirl1908 with https://t.co/A97yVpOwpW
* @LisaCrispin with DevTestOps https://t.co/XFIoasi9rJ
* @FriendlyTester with https://t.co/2afqq8c8pz @ministryoftest

— T.J. Maher (@tjmaher1) May 30, 2019

... Make sure to read what my co-worker, Carl Robinson has been doing with Docker!

Happy Testing!

-T.J. Maher
Sr. QA Engineer, Software Engineer in Test
Meetup Organizer, Ministry of Testing - Boston

Twitter | YouTube | LinkedIn | Articles

Check out what the Ministry of Testing, Applitools + Test Automation University, and Mabl + Lisa Crispin have brewing!

There is so much going on in the software testing community with the Ministry of Testing, Applitools' Test Automation University, and Mabl and Lisa Crispin's webinar series, Modeling Your Test Automation Strategy!

Basic Capybara-Gauge: Set Up Headless Chrome

This is the fourth part of a series of blog posts. Care to go back to the beginning? 

We've come up with test specs for Dave Haeffner's The-Internet. We've set up a Ruby environment to run the tests, and configured Chrome to open the Login page.

With this entry, we will be activating Chrome in Headless Mode, i.e. running Chrome but without a visual interface, allowing the tests to run faster.

About Headless Chrome

According to Eric Bidelman's article, Getting Started with Headless Chrome, the new option of running Chrome in Headless mode started with Google Chrome version 59, released in May 2017.

Before, you had to use something such as PhantomJS to emulate the Chrome browser if you wanted to run Chrome in a place where the processing power to bring up a visual window would be a waste, such as running it with an automated test script running in a continuous integration environment. Now, it is built right in.

Basic Chef + Test Kitchen: Setting up Ubuntu with Vagrant + Virtual Box + the Test-Kitchen Ruby Gem on a MacBook

Let's say we wanted to test something out on Ubuntu, a popular Linux distribution, on a MacBook: How would we be able to set it up locally, configuring the workstation? We set it up in Docker before. What other ways are there?

One way to do it is with a Chef.io tool to spin up a test environment, Test-Kitchen, and its Ruby gem. And, yes, this cooking metaphor is extended all throughout the Chef toolset with its cookbooks and recipes for environment configurations.

Adventures in Automation started looking at The Ruby Behind Chef, the History Behind Test Kitchen, and exploring the free tutorials that Chef provides, the extensive Learn Chef Rally.

For this article we will be using:

• Test-Kitchen on GitHub as a primary source: builds a Vagrantfile, allowing you to create or destroy the instances of your virtual machines
• Vagrant: "a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the "works on my machine" excuse a relic of the past".
• Oracle's VirtualBox: "VirtualBox is a cross-platform virtualization application. What does that mean? For one thing, it installs on your existing Intel or AMD-based computers, whether they are running Windows, Mac, Linux or Solaris operating systems. Secondly, it extends the capabilities of your existing computer so that it can run multiple operating systems (inside multiple virtual machines) at the same time."

Setting up Ubuntu Linux with Docker

Let's say that I only have a Windows 10 system at home, and I wanted to take a refresher course, Michael Hartl's Learn Enough Command Line to Be Dangerous. There are many ways to get Linux on a Windows machine. The easiest I found? Docker. Besides, they use Docker at work, so I need to practice that, too.

There are some technologies that keep on popping up again and again in my short career as an automation developer. One of those is Docker. "Docker is a software platform that allows you to build, test, and deploy applications quickly. Docker packages software into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime", according to AWS's article, What is Docker?

Back in July 2016, I was doing some weekend work to see if my company might be able to set up Docker with the new Docker Toolbox, with me working with our DevOps team, implementing a Selenium Grid using Docker-Selenium to possibly offload some of the cost of Sauce Labs, which can get pricey. I was inspired by a Selenium Conf India 2016 talk, "Testing as a Container". The month before, I was also tinkering with Setting up a Virtual DEV environment with VirtualBox, Vagrant and Docker, based on a Test Driven Java Development book I was reading. The projects were nixed, and I moved onto other things.

Years later, in March 2018, I would revisit this project, Starting a Selenium Grid using AWS + SeleniumHQ Docker images and Docker Compose.

That's how it goes jumping from gig to gig the past few years: from workplace to workplace the technology stack changes. Hopefully with my current full time permanent gig I can stop tinkering, settle down for five or six years and actually get good at some of this stuff, instead of always feeling that I am paging through the book "How To Skydive" while I am in free fall.

With this blog article we are not doing anything so fancy. All I want to do is bring up a Docker image containing Ubuntu. You see, as much as I love my work Macbook, at home I always use a Windows machine. Every few months I am tinkering with a new tool or technology, installing the Windows version of whatever I am tinkering with. It gets cluttered with half-forgotten tools. Maybe if I use Docker, I can start using a real Linux environment while at home instead of the half-baked hybrid I have been using.

Yeah, I could Start a Linux Machine with Amazon Web Services Free Tier, which I researched back in January of 2018 ... but between you and me, I worry about leaving a process running, and being stuck with a hefty monthly bill. Luckily, Docker for Windows would work just as easily!

Docker Community Edition

Starting a Selenium Grid using AWS + SeleniumHQ Docker images + Docker Compose

With this blog post we will be exploring how to start a Selenium Grid using an Amazon Web Service (AWS) instance the SeleniumHQ Docker images, Docker and Docker Compose. For this example, I will be using a Macbook, so the Terminal will be my Command Line Interface.

Need more information? Check out the official Amazon Web Services documentation:

• What is Docker?
• What is an Amazon Machine Image (AMI)
• Amazon Linux AMI FAQ
• Docker overview

We could use the "Amazon Elastic Container Service" and using AWS Fargate to run containers without managing servers or clusters ... but since I am tinkering, I want to make sure that I am using only free services provided by the Amazon Free Tier that I signed up for last month.

We are going to be starting off by following along with the official docs by Amazon, AWS Docker Basics.

Introduction to the Learn Chef Rally!

Need to automate configurations for Amazon Web Services? One way to do it is with Chef.io.

Chef does the "cooking" metaphor to the hilt! Configurations are "recipes", written in the Ruby programming language, which are stored in a "cookbook". We covered this in early January with The Ruby Behind Chef.

This metaphor continues: To automate all of this, you use the product "Test Kitchen".

Seems complicated? To help you out, Chef create the most amazing set of documentation I have ever seen: The Learn Chef Rally.

From Getting Started With Chef in Learn.Chef.io:

"If you're new to Chef, just know that Chef lets you automate all the things—infrastructure, applications, compliance and more. Learn Chef Rally will teach you how. It's the hub for Chef and DevOps related resources that will build your skills.

"The Learn Chef Rally curriculum is organized by tracks. A track groups related learning activities.

"We call each learning activity a module. Modules include a mix of hands-on labs, articles, and videos. Our hands-on labs help you get started quickly on the platform that you care about most.

"When you finish a module, you'll see a few quiz questions at the end. Answering these questions correctly gives you credit for having completed the module. You can track your progress on the home page".

Since my job will be automating Amazon Web Service tests, spinning up and tearing down environments, I'm starting the module on Test Kitchen: "Infrastructure Code Deserves Tests Too".

Starting a Linux Machine with Amazon Web Services Free Tier

It's my first time working at a cloud-security company using Amazon Web Services (AWS)! How to get up to speed?

To get up to speed, first, I've attempted to decipher the AWS Alphabet Soup in an earlier blog post. Next, to get more exposure to AWS, I signed up for the AWS Free Tier. I was a bit leery, since I needed to punch in my credit card information when I signed up.

• Some AWS services are free for 12 months, provided I do not go over 750 hrs per month of usage of EC2, and over 5 GB of Storage in Amazon EFS or 30 GB of Elastic Block Storage for long term. If I go over, or accidentally use a service that isn't on the free tier, I worry I am going to rack up a huge charge.
• Some services seem to be always free. 

Luckily, I didn't have to look far in order to find help!

• Getting Started Resource Center: Mentioned in the email received upon sign up.
• AWS Documentation 
• AWS 10 Minute Tutorials on everything that you could think of!

Top Adventures in Automation blog entries for 2017

With Adventures in Automation, How Was 2017?

So far, in 2017, this blog attracted the attention of 33,700 different users, according to Google Analytics.

Here are the top five blog posts for Adventures in Automation for 2017:

#5: What happens when you ask the Twitterverse: "How Can a Manual Tester Switch To Automation?"

#4: Build a Basic Appium Framework: Review How to Inspect Mobile Apps with Appium Desktop

#3: Learning Serenity BDD: An Automation Framework That Uses Specification by Example (SBE)

#2: New automation framework released to test APIs: Introducing Karate, by Intuit India's Peter Thomas 

#1: Learning Appium Desktop: What is Appium Server and How Do You Start It Through Appium Desktop?

Thank you, dear reader, for all your continued support.

Coming in 2018:

New Blog Entries! 

Mid-January 2018, I start a new job as a Software Development Engineer in Test: An SDET role at a security company, tackling brave new technologies -- some I've only tinkered with, some which are brand-new to me.

I have done API testing before, using Apache Http Components, Rest Assured, and Twitter4J. Most of these have been in Java, though. Although I have tinkered with Docker and Vagrant, I have never used DevOps tools such as Chef or Kitchen. And I have never used Scala or Ruby, two of the primary coding languages they will be using.

... I made sure to tell my future employer all of this -- I am nothing if not brutally honest on job interviews. They hired me anyway, saying they would provide plenty of on-the-job training. Not wanting to embarrass myself, I've started to hit the books. There are plenty of resources on Scala, Chef, and even the API of the product I will be testing.

Expect many more blog entries on these subjects in the coming months as I do some extra-curricular learnin'.

New Speaking Engagements! 

I am a speaker at Joe Colantonio's Automation Guild 2018, and will be giving a live Q & A session January 10th, 2018 at 11 AM.

https://youtu.be/Ja7tCXguXXw

A New YouTube Channel!

I will be starting a new weekly video blog, sharing my experience with software testing, starting Monday, January 1st, 2018. Subscribe now!

I hope you and yours have a happy and healthy 2018!

Happy Testing!

-T.J. Maher
Twitter | LinkedIn | GitHub

// Sr. QA Engineer, Software Engineer in Test, Software Tester since 1996.
// Contributing Writer for TechBeacon.
// "Looking to move away from manual QA? Follow Adventures in Automation on Facebook!"