Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report

So far, we have reviewed how to review code with ES Lint + Prettier and Typecheck, how to set up and run smoke tests, how to run all Playwright + TypeScript tests, and reviewed the HTML report of results.

In this post, we are going to set up a three stage GitLab CI/ CD pipeline that will run against every merge request: 

•  Quality (lint, typecheck, prettier)  --> Test ( smoke + regression ) --> Report ( Downloadable )

I haven't used GitLab since when I worked at ThreatStack back in 2020. ( See my blog entry Getting to Know GitLab and How They Test the UI )

GitLab reads .gitlab-ci.yml from the repository root and turns it into a pipeline, a set of jobs, grouped into stages, running whenever you push a change to a repository. 

Stages run one after another, and jobs within a stage run in parallel.  If one stage fails, the stages after it are skipped. 

This begs the question: What is GitLab? What is CI/ CD? Or a pipeline? Or a merge request? 

How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json

Earlier, we went over how we could create scripts in the package.json file of our Playwright framework to add typechecking, linting, and formatting your code with prettier. 

• Want to see the entire bun-create-playwright project? It is on tinyurl.com/bun-create-playwright!

With this post, we will explore how the built-in test runner for Playwright Test can run headed tests, debug versions of tests, and smoke tests.

... and shortcuts for all of these can be set up in the scripts in your package.json! If you are using "bun" as a package manager, as we are in bun-create-playwright, just type out "bun run", a space and then the shortcut such as: bun run test

package.json

"scripts": {

    "test": "playwright test",

    "test:headed": "playwright test --headed",

    "test:trace": "playwright test --trace on",

    "test:chromium": "playwright test --project=chromium",

    "test:firefox": "playwright test --project=firefox",

    "test:webkit": "playwright test --project=webkit",

    "test:smoke": "playwright test --project=chromium --grep '@smoke'",

    "test:flaky": "playwright test --project=chromium --repeat-each=20",

    "test:ui": "playwright test --ui",

    "test:debug": "playwright test --debug",

    "test:failed": "playwright test --last-failed",

    "test:login": "playwright test tests/login.spec.ts",

    "test:secure-area": "playwright test tests/secure-area.spec.ts",

    "report:list": "playwright test --reporter=list",

    "report:line": "playwright test --reporter=line",

    "report:dot": "playwright test --reporter=dot",

    "report:blob": "playwright test --reporter=blob",

    "report": "playwright show-report",

    "codegen": "playwright codegen",

    "lint": "eslint .",

    "lint:ci": "eslint . --max-warnings 0",

    "lint:fix": "eslint . --fix",

    "format": "prettier --write .",

    "format:check": "prettier --check .",

    "format:debug": "prettier --check . --log-level debug",

    "format:diff": "prettier --list-different",

    "typecheck": "tsc --noEmit"

  },

bun-create-playwright / package.json

Do you need to really set up shortcuts like these? Certainly not! But it is easier than typing out: bunx playwright test --project=chromium --grep '@smoke'.

Feel free to name these commands anything you want! 

Implementing Page Objects in Playwright

Picture a login screen, such as The-Internet / Login. 

On this LoginPage, there is:

• a heading: Login Page
• a user name textbox with the label, "Username"
• a password textbox with the label, "Password"
• a login button, with the role of a button, and the name of "Login"
• a flash message that appears if you enter invalid credentials such as "NotAUser" and "NotAPassword".

If you successfully log in with "tomsmith" and "SuperSecretPassword!, there is a SecureArea:

• a heading: "Secure Area"
• a flash message "You logged into a secure area!"
• a Logout button. 

Sure, you could interact with each web element in your test... but what if the username text box locator changes? You would have to update multiple tests every time the element changed. 

... Instead, you could place it in a Page Object, something that Playwright handles well!

"A page object represents a part of your web application. An e-commerce web application might have a home page, a listings page and a checkout page. Each of them can be represented by page object models.

"Page objects simplify authoring by creating a higher-level API which suits your application and simplify maintenance by capturing element selectors in one place and create reusable code to avoid repetition".

Using Playwright's Built-In Test Runner? Or Something Else?

You may have noticed in https://playwright.dev/docs/pom that there are two different styles of page objects. One for "Test". One for "Library". 

• Test: If you are writing actual Playwright test suites, and Playwright's built in test runner, use the Test section as a guide when creating page objects. 
• Library: If you are integrating Playwright into an existing test framework such as Jest or Cucumber and just want browser automation, instead of having pre-built page fixtures, etc, you can use this format. 

How Playwright Frameworks get configured with playwright.config.ts

When we installed bun, a new package manager owned by Anthropic, then ran "bun create playwright", a new automation framework was stood up, along with sample tests, and a Playwright configuration file. In this post, we will be examining the file generated: playwright.config.ts.  

Personally, I find the pre-generated file very hard to scan... there are so many options and documentation in the comments, it is difficult for me to focus on the code. Let's examine just the code generated below. If you need to see the whole file, you can see it here: https://playwright.dev/docs/test-configuration

Playwright.dev / Configuration mentions, "Playwright has many options to configure how your tests are run. You can specify these options in the configuration file". 

Running Tests with Playwright Test Explorer and Generating Tests with Codegen

Finding the best locator for a web element can be a hassle:

• Right clicking on an element in Google Chrome. Inspecting the element. Going to Chrome Developer Tools. Try to decide what to do if there isn't a clear test id. 

Playwright comes with a built in code generator where it can built out a rough draft of a test while you interact with a website. "Playwright will look at your page and figure out the best locator, prioritizing role, text and test id locators. If the generator finds multiple elements matching the locator, it will improve the locator to make it resilient that uniquely identify the target element", according to Playwright.dev / Test Generator.

Do you have the Integrated Development Environment (IDE) by Microsoft, VS Code? You can get it at the Visual Studio Marketplace. 

Playwright Test Explorer

• Playwright.dev / Getting Started VS Code: https://playwright.dev/docs/getting-started-vscode

After installation, you will see a beaker icon in your VS Code left navigation menu. Selecting that, you can see your tests, such as the default ones Playwright automatically adds when it is installed: 

Checking code with lint, formatting it with prettier

Now that we've installed bun, Anthropic's package manager, scaffolded a Playwright framework and closely examined the results, and added typechecking with TypeScript's compiler, it's time to add ways to check the code with lint, and reformat the code with prettier. 

We will be using:

• ESLint, as the static-analysis tool to review the code without running it. Little bits of fluff -- like syntax errors, structural bugs, anti-patterns, and code style violations -- can collect on your code, so it helps to run a linter to help catch it all, such as ESLint. There is also a linter,  eslint-plugin-playwright, for Playwright tests.
• Prettier enforces a consistent code style across your entire codebase. Because ESLint and Prettier can conflict, we will be using Prettier's eslint-config-prettier, which turns off all rules that are unnecessary or might conflict with ESLint.

What is a Linter? 

According to the Wikiwand entry for Lint, "Stephen C. Johnson, a computer scientist at Bell Labs, came up with the term 'lint' in 1978 while debugging the yacc grammar he was writing for C and dealing with portability issues stemming from porting Unix to a 32-bit machine. The term was borrowed from lint, the tiny bits of fiber and fluff shed by clothing, as the command he wrote would act like a lint trap in a clothes dryer, capturing waste fibers while leaving whole fabrics intact. The lint program was released outside of Bell Labs in Unix V7, in 1979.

"In his 1978 paper, Johnson explained his reasons for creating a new program to detect errors: '...the general notion of having two programs is a good one' because they concentrate on different things, thereby allowing the programmer to 'concentrate at one stage of the programming process solely on the algorithms, data structures, and correctness of the program, and then later retrofit, with the aid of lint, the desirable properties of universality and portability' "

Add Type Checking and TSConfig to Bun-Create-Playwright

Now that we have installed bun, Anthropic's package manager, and scaffolded a Playwright framework and closely examined the results, we are going to explore with our Bun-Create-Playwright project ways to check if our code is correct. 

The first method we will be exploring is typechecking. 

Why Typechecking? As the Playwright.dev / Node.js Introduction mentions:

"[...] Playwright does not check the types and will run tests even if there are non-critical TypeScript compilation errors. We recommend you run TypeScript compiler alongside Playwright.

"[...] Note that Playwright only supports the following tsconfig options: allowJs, baseUrl, paths, references and extends.

"[...] By default, Playwright will look up a closest tsconfig for each imported file by going up the directory structure and looking for tsconfig.json or jsconfig.json. This way, you can create a tests/tsconfig.json file that will be used only for your tests and Playwright will pick it up automatically".

Before we go further down this road...

What is Type Checking? 

According to Type Checking in TypeScript: A Beginners Guide, every piece of data in TypeScript is given a "type", and this "type" determines what properties the data has, and what methods are available to it. The types can be things like a Number, String, Enum, Boolean, Array, Object, Type assertions, or others.

What happens when you scaffold a Playwright framework and run installed tests using bun?

We covered in the last blog post how to install bun, a new package manager. 

In this blog post, we will look into using bun to install a new Playwright framework. 

Create the Playwright Framework

Once I installed bun on my Windows PC and was up and running I created a new folder, "bun-create-playwright".

I opened up that folder in VS Code, along with a new PowerShell terminal. In that terminal I entered:

• bun create playwright 

This activated the interactive Playwright installer that Playwright comes with. 

• I selected I wanted it to create for me a TypeScript project, placing the tests in the default folder, tests, but I decided not to add a GitHub Actions workflow. I wanted to experiment with using GitLab. 
• I chose it to set up and install all the browsers for me... which it did... using NPX, part of Node.

Wait a second, when it comes to our Playwright framework, doesn't bun replace node?

No. Bun might be a JavaScript runtime, a package manager, and a bundler shipped as a single software tool, but in our case we are simply using it as a package manager in our Playwright framework. 

Bun works alongside Node.js in the Playwright project. We still use Playwright's test runner, a Node.js program. Bun handles installations and run tasks.

Introducing bun, a new package manager and JavaScript runtime environment

New job? New toolset to explore! In this post I'll be investigating a new JavaScript package manager and runtime environment called bun.sh. 

You may have noticed on Playwright.Dev's Getting Started / Installation section there are three different ways to install a ui or api test automation framework with the latest version of Playwright:

• npm init playwright@latest
• yarn create playwright
• pnpm create playwright

For this blog post I'll be investigating a fourth way, the new toolset they just started using at work, bun.sh.

• bun create playwright

What is npm, yarn, pnpm, and bun? These are called package managers.

What is a package? Developing a project, you don't have to figure out how to code everything yourself. You can include in your projects outside libraries, or "packages" of code from from the public JavaScript registry npmjs.com. 

The problem is that these packages then need to be downloaded, installed, managed, and updated whenever they are updated. JavaScript projects can use a package manager, such as bun, to help this process. 

When putting together a Playwright automation framework based around JavaScript and TypeScript, Playwright uses the Node.js JavaScript runtime environment, to run its test runner, execute automation scripts, and manage its browser instances.(See Nodejs.org / About )

There is a division of labor: Node.js runs these internal functions like running tests. NPM, Yarn, PNPM, and now Bun handle the packages. 

Why are there so many package managers? Each has its own specialization. The original NPM is bundled already with Node.js. Yarn is a Facebook toolset that was created when NPM was found to work too slow. PNPM was created to be smaller and therefore faster. And bun was built for quick startup and installation speed, good especially if you have a CI/ CD pipeline that has a LOT of packages to install each time the tests run.

Does giving a presentation via Zoom to the Sydney Testers Meetup make me an international speaker?

If I presentation to the Sydney Testers Meetup in Australia, from my home in the Massachusetts, through Zoom, does that make me an international speaker? [ Vote in the New Poll, on LinkedIn ]

There is still time to sign up for the event at: https://www.meetup.com/sydney-testers/events/315166776


"T.J. Maher, the former organiser at Ministry of Testing Boston, blogger and speaker, will be presenting from the US on using the Detox framework to create effective test automation frameworks for React Native applications.