You may have noticed that in the GitLab CI / CD Pipeline for our Playwright project, in the .gitlab-ci.yml file, after our scripts has run, in the Test stage, there is a subsection called "artifacts" with certain paths to something called "playwright-report", "test-results", and "reports".
The Playwright -> Artifacts -> Reports stage
playwright:
extends: .bun_playwright
stage: test
timeout: 30 minutes
...
...
artifacts:
when: always
paths:
- playwright-report/
- test-results/
- reports/
reports:
junit: reports/junit/results.xml
expire_in: 30 days
Every time Playwright tests run, it generates proof of the test execution: screenshots capturing the state of the UI when a test failed, video recordings of an entire browser session, HTML and JUnit XML test reports, trace reports showing action logs, network requests. These artifacts, when the tests run locally, are placed by the Playwright Test runner in generated folders, playwright-report and test-results, along with a folder called reports/junit.
With this post, we will be examining how this artifacts stage of the GitLab CI/ CD pipeline produces downloadable artifacts we can examine.
Playwright-Report: A Playwright Test folder
The Playwright Test runner creates a folder called "playwright-report", if it doesn't already exist, and an HTML file called index.html documenting the completed test run.
From Playwright.dev / Test Reporters: "Playwright Test comes with a few built-in reporters for different needs and ability to provide custom reporters. [...] All built-in reporters show detailed information about failures, and mostly differ in verbosity for successful runs. [...] HTML reporter produces a self-contained folder that contains report for the test run that can be served as a web page".
The HTML Report is the one that automatically opens after you run your tests locally, or if you are using the node package manager: npx playwright show-report ... or if you are using bun: bunx playwright show-report.
By default, this report is written into the playwright-report folder.
Test-Results: A Playwright Test folder
The test-results folder is the default folder for all output artifacts such as:
- screenshots
- videos
- traces captured on failure or retry
Videos: You can explicitly tell Playwright Test to record videos either for each test ("on"), record a video for each test but delete the video if the test is successful, ("retain-on-failure"), or record video only when retrying a test for the first time ("on-first-retry") - From Playwright.dev / Videos
Traces: See Playwright.dev / Trace Viewer. Locally, you could run a test with: bunx playwright test --trace on
Reports: Contains Playwright's generated JUnit report. We are telling GitLab to take the reports/junit/reselts.xml file, zip it up, and store it so later we can turn it into a downloadable artifact.
What is a GitLab Artifact?
GitLab Docs, under "Job Artifacts" says, "Jobs can output an archive of files and directories. This output is known as a job artifact. Artifacts can include build output or report files. By default, later jobs fetch a copy of all artifacts from jobs in earlier stages.
"For example, an early job can build a project and save the output as an artifact. Then a later job fetches the artifact and runs tests on the saved build output. For a full list of supported configuration for the artifacts keyword, see the GitLab CI/CD YAML syntax reference.
"To create job artifacts, use the artifacts keyword in your .gitlab-ci.yml file
"[...] The paths keyword determines which files to add to the job artifacts. All paths to files and directories are relative to the repository where the job was created".
So, the artifacts block we listed will create three directories:
- playwright-reports
- test-results
- reports
- For now, we have indicated we want to expire the artifacts in 30 days.
What is GitLab's Artifact:Reports?
This folder is where we are going to be storing the Playwright generated JUnit XML report of the test results.
Under GitLab CI/CD artifacts report types, "Use artifacts:reports to:
- "Collect test reports, code quality reports, security reports, and other artifacts generated by included templates in jobs.
- "Some of these reports are used to display information in: Merge requests. Pipeline views. Security dashboards.
What is Artifact:Reports:JUnit?
"The junit report collects JUnit report format XML files. The collected Unit test reports upload to GitLab as an artifact. Although JUnit was originally developed in Java, there are many third-party ports for other languages such as JavaScript, Python, and Ruby".
The playwright/reports/junit/results.xml file is a machine-readable test report structured in the standardized JUnit XML format, capturing high-level execution statistics and detailed test suite metadata so that Continuous Integration (CI/CD) pipelines can natively parse and display test results, metrics, and trends.
The file wraps your tests into <testsuites> and <testsuite> elements, with individual tests mapped to <testcase> tags.
<testsuites id="" name="" tests="9" failures="0" skipped="0"
errors="0" time="5.402829">
<testsuite name="login.spec.ts" timestamp="2026-08-29T16:29:20.251Z"
hostname="chromium" tests="2" failures="0" skipped="0" time="7.657"
errors="0">
<testcase name="Login › unsuccessful login with invalid credentials"
classname="login.spec.ts" time="3.828">
</testcase>
<testcase name="Login › successful login with valid credentials"
classname="login.spec.ts" time="3.829">
</testcase>
</testsuite>
<testsuite name="secure-area.spec.ts" timestamp="2026-08-29T16:29:20.251Z"
hostname="chromium" tests="1" failures="0" skipped="0" time="3.963" errors="0">
<testcase name="Secure Area › logout returns to login with success alert"
classname="secure-area.spec.ts" time="3.963">
</testcase>
</testsuite>
So, the artifacts stage in in our test run stores all the artifacts in our test run.
Now that the artifacts have been placed, the reports can be run!
The Reports Stage
Until then, Happy Testing!
Assemble the Reports Stage!
In the Gitlab CI/ CD pipeline:
- The Quality stage checked the code before the tests ran.
- The Test stage ran the Playwright test and handled the artifacts.
- The Report stage assembles the report.
The Reports Stage
report:
stage: report
image: alpine:3.24
# Runs even when the test job failed, which is when the report matters most.
when: always
needs:
- job: playwright
artifacts: true
optional: true
- job: playwright:manual
artifacts: true
optional: true
script:
- ls -la playwright-report/ test-results/ reports/junit/ || true
artifacts:
when: always
paths:
- playwright-report/
- test-results/
- reports/
reports:
junit: reports/junit/results.xml
expire_in: 30 days
Here, we are using a lightweight Docker image called alpine, an Alpine Linux 3.24 container to execute this stage. A heavy image with Node/ Playwright isn't needed.
This stage will start up as soon as the "playwright" job in Tests stage finishes. It will always start whether or not the other stages fail.
- The earlier playwright job? It needs those artifacts we set up.
- It doesn't matter if the playwright job was kicked off by a merge (the playwright job) or a new pipeline created manually (the pipeline:manual job we set up earlier).
For visibility's sake, we will list out in the logs all the contents of playwright-report, test-results, and reports/junit. If there are no contents, that is fine. || true ensures the job does not fail if one of those directories does not exist.
With the paths, we set up playwright-report, test-results, and reports/junit as downloadable artifacts.
And in reports, this file isn't just archived. It is parsed, and puts the results directly in GitLabs's merge request UI, showing all pass/ fail counts.
Now, everything can be downloaded and examined to see how the test run went!
Next, we can look into posting the HTML report onto GitLab Pages.
Bun Create Playwright:
- Part One: Introducing bun, a new package manager and JavaScript runtime environment
- Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
- Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
- Part Four: Checking code with lint, formatting it with prettier
- Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
- Part Six: How Playwright Frameworks get configured with playwright.config.ts
- Part Seven: Implementing Page Objects in Playwright
- Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
- Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
- Part Ten: How Developers Can Test Their Feature Branch Against Various Playwright Configurations Before Creating a Merge Request in GitLab CI/CD
- Part Eleven: Examining the artifacts job of a Playwright GitLab CI / CD pipeline
- Part Twelve: Publishing Playwright HTML reports on GitLab Pages
- GitLab: https://gitlab.com/tjmaher/bun-create-playwright
No comments:
Post a Comment