Developers need the option to test out their feature branch before merging it into main. In the last section, we set up a GitLab CI/CD Pipeline in our bun-create-playwright project to check every GitLab Merge Request.
Now, we will be giving developers option to create a new pipeline where they can choose:
- Which feature branch to test against.
- Which test suite to execute (LoginPage tests? Secure Area? Or all of them?)
- Which Playwright browser to use: Chromium, Firefox, WebKit, or all.
All these features can be set up using our .gitlab-ci.yaml file!
Go to to bun-create-playwright, view the Pipelines section, and select the New Pipeline button.
Run a New Bun-Create-Playwright Pipeline
If you go to the Run new pipeline page at https://gitlab.com/tjmaher/bun-create-playwright/-/pipelines/new you can see the following default values I've set:
By default:
- Branch name is "main", TEST_SUITE is "all", BROWSER is "all".
... But all these values can be set, with the developer choosing the options laid out in the .gitlab-ci.yaml file.
Collect User Data With Spec Inputs
We are going to be adding to the GitLab Pipeline we built in our last post Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report.
First, we are going to add a "spec" section to the header of the YAML file to define the behavior of a pipeline, and within that spec header, we will ask the person who triggered a new pipeline to set user defined inputs for the pipeline.
spec:
inputs:
browser:
description: "Browser project to run against."
default: "all"
options: ["all", "chromium", "firefox", "webkit"]
suite:
description: "Select full regression suite, or a single spec"
default: "all"
options: ["all", "login.spec.ts", "secure-area.spec.ts"]
This is where we declare the "browser" and "suite" variables that appear when a user creates a new pipeline.
We've given them a choice to run their feature branch code against:
- Browser: All, Chromium, Firefox, or Webkit.
- Suite: A full regression with all tests to be run? Just the login.spec.ts ones we set up? Or just the secure-area.spec.ts ones?
From Playwright.dev / Browsers: "Each version of Playwright needs specific versions of browser binaries to operate. You will need to use the Playwright CLI to install these browsers.
"With every release, Playwright updates the versions of the browsers it supports, so that the latest Playwright would support the latest browsers at any moment. It means that every time you update Playwright, you might need to re-run the install CLI command.
"With every release, Playwright updates the versions of the browsers it supports, so that the latest Playwright would support the latest browsers at any moment. It means that every time you update Playwright, you might need to re-run the install CLI command.
"[...] Playwright can run tests on Chromium, WebKit and Firefox browsers as well as branded browsers such as Google Chrome and Microsoft Edge. It can also run on emulated tablet and mobile devices. See the registry of device parameters for a complete list of selected desktop, tablet and mobile devices.
"[...] Playwright's WebKit is derived from the latest WebKit main branch sources, often before these updates are incorporated into Apple Safari and other WebKit-based browsers. This gives a lot of lead time to react on the potential browser update issues. Playwright doesn't work with the branded version of Safari since it relies on patches. Instead, you can test using the most recent WebKit build.
Add New Workflow Rules
Next, we will add a few new rules to the workflow.
"The workflow keyword is evaluated before jobs. For example, if a job is configured to run for tags, but the workflow prevents tag pipelines, the job never runs".
According to the Gitlab Docs / yaml / workflow, here are some example if clauses for workflow rules:
| Example rules | Details |
|---|---|
if: '$CI_PIPELINE_SOURCE == "merge_request_event"' | Control when merge request pipelines run. |
if: '$CI_PIPELINE_SOURCE == "push"' | Control when both branch pipelines and tag pipelines run. |
if: $CI_COMMIT_TAG | Control when tag pipelines run. |
if: $CI_COMMIT_BRANCH | Control when branch pipelines run. |
What Are GitLab Pre-Defined Variables?
Before you create a GitLab pipeline, just after you create a pipeline, certain pre-defined variables (See GitLab Docs) get filled in with values describing what type of pipeline is created. For example:CI_PIPELINE_SOURCE: Pre-pipeline
- How the pipeline was triggered. The value can be one of the pipeline sources
- What triggered the pipeline? It can capture values such as: api, chat, external, external_pull_request_event, merge_request_event, ondemand_dast_scan, ondemand_dast_validation, parent_pipeline, pipeline, push, schedule, security_orchestration_policy, trigger, web, webide
CI_COMMIT_BRANCH: Pre-pipeline
- The commit branch name. Available in branch pipelines, including pipelines for the default branch. Not available in merge request pipelines or tag pipelines.
- Available in branch pipelines, including pipelines for the default branch. Not available in merge request pipelines or tag pipelines.
CI_OPEN_MERGE_REQUESTS: Pre-pipeline
- A comma-separated list of up to four merge requests that use the current branch and project as the merge request source.
- Only available in branch and merge request pipelines if the branch has an associated merge request. For example, gitlab-org/gitlab!333,gitlab-org/gitlab-foss!11
Set Up Rules How Pipelines Get Triggered
Here, we are setting up some rules on how pipelines get triggered, depending on what triggered them.
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "web"
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
stages:
- quality
- test
- report
- Pipeline started manually via the web? Run the pipeline immediately.
- Triggered by a Merge Request event creating an MR or pushing a fix to an open MR? Run the pipeline as a detached merge request pipeline.
- Is this a regular branch commit AND does the branch have an open merge request? Well, we don't want two pipelines spun up. Don't run the branch. The MR branch will handle it.
- Is this a standard commit push to any branch, and does it clear the previous rule? Run a regular branch pipeline.
Define Two Templates: Bun_Quality and Playwright
Like earlier, we are building out two GitLab CI/ CD stages:
- .bun_quality and .bun_playwright
The dot "." before the name hides the stages, making them to be good when trying to set up templates that can be followed when setting up what needs to happen before the script in that stage is run.
.bun_quality:
image: oven/bun:1.3.14
before_script:
- bun --version
- bun install --frozen-lockfile
.bun_playwright:
image: mcr.microsoft.com/playwright:v1.62.1-noble
before_script:
- apt-get update && apt-get install -y unzip curl
- curl -fsSL https://bun.com/install | bash -s "bun-v1.3.14"
- export PATH="$HOME/.bun/bin:$PATH"
- bun --version
- bun install --frozen-lockfile
The .gitlab-ci.yml file then sets up:
- Quality stage to run format, lint, and typecheck, while the Playwright
... But the stage running the Playwright tests will be slightly different.
Set Up New Stage When Pipeline is Manually Triggered
Running Playwright tests will be slightly different between one that was kicked off by pushing code to the repository and one that was manually kicked off when creating a new pipeline.
If a developer went to Pipelines -> New Pipeline on the web, we do NOT want this stage run.
playwright:
extends: .bun_playwright
stage: test
timeout: 30 minutes
rules:
- if: $CI_PIPELINE_SOURCE == "web"
when: never
- when: on_success
script:
- bun run test:smoke
- bun run test
artifacts:
when: always
paths:
- playwright-report/
- test-results/
- reports/
reports:
junit: reports/junit/results.xml
expire_in: 30 days
We want a whole new stage to be run, one where we gather the inputs the user has defined and given value.
playwright:manual:
extends: .bun_playwright
stage: test
timeout: 30 minutes
rules:
- if: $CI_PIPELINE_SOURCE == "web"
when: on_success
- when: never
variables:
BROWSER: "$[[ inputs.browser ]]"
SUITE: "$[[ inputs.suite ]]"
Although both have the same .bun_playwright setup, the playwright one will run all smoke then all tests on all browsers. The playwright:manual will be gathering as variables the inputs.browser and inputs.suite the developer selected.
The playwright one will exclude anything triggered by the web as a rule, the one we set.
The playwright: manual one will exclude anything that is not manual.
Add Collected Variables To The Script Running Tests
Now that we have collected all the variables needed from the user, we can start using them to execute our test scripts.
script:
- |
PROJECT_ARG=""
if [ "$BROWSER" != "all" ]; then
PROJECT_ARG="--project=$BROWSER"
fi
SUITE_ARG=""
if [ "$SUITE" != "all" ]; then
SUITE_ARG="tests/$SUITE"
fi
echo "Running: bun run test -- $PROJECT_ARG $SUITE_ARG"
bun run test -- $PROJECT_ARG $SUITE_ARG
- If the user selected "all" browsers and "all" suites, it will kick off all tests, running them against the feature branch selected, with the command: bun run test.
- If the user selected "chromium" and "login.spec.ts", then that will be passed into the command: bun run test -- --project=chromium --tests/login.spec.ts
This way, a developer can test out their feature branch before they put in a merge request.
Next, we will be examining how to publish the reports on GitLab Pages.
Bun Create Playwright:
- Part One: Introducing bun, a new package manager and JavaScript runtime environment
- Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
- Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
- Part Four: Checking code with lint, formatting it with prettier
- Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
- Part Six: How Playwright Frameworks get configured with playwright.config.ts
- Part Seven: Implementing Page Objects in Playwright
- Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
- Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
- Part Ten: How Developers Can Test Their Feature Branch Against Various Playwright Configurations Before Creating a Merge Request in GitLab CI/CD
- GitLab: https://gitlab.com/tjmaher/bun-create-playwright
Until then, Happy Testing!
-T.J. Maher
Software Engineer in Test
BlueSky | YouTube | LinkedIn | Articles
No comments:
Post a Comment