If you want to do some basic visual checking to see if there has been any deviation from your baseline of images, you can use Playwright. It's built in! Playwright can take a snapshot of a web element, a visible viewport, or a full page, and save it in your Git repository as a baseline, failing the test if the image, page, or viewport does not match up.
Caution: From what I have been reading, this can quickly cause your code repository to balloon in size, since Chrome, Firefox, and WebKit would each store its own golden screenshot in your repo. Also, images on Mac, Windows, and Linux all appear different pixel-by-pixel. If using a CI platform, it might be best to run visual tests only on a standard Playwright Docker image, to generate and compare snapshots. According to TestQuality, "Once a suite passes 50–100 visual tests, teams need a layer that tracks run history, surfaces flaky-test patterns across cycles, and routes confirmed defects into the team's tracker — none of which lives inside the test runner itself"... I wonder if you can store images in an Amazon S3 bucket and hook that up as a virtual drive? ... no matter. That will be a blog post for another time...
Right now, we will be walking through Butch Mayhew's code he wrote for his LinkedIn Learning course, Learning Playwright, found on his companion GitHub site.
While the test is in a certain state you can a screenshot of the page or certain elements of the page and save them as a snapshot. The snapshots an be used as a baseline images to compare your current site against. This baseline can be periodically updated as the site evolves.
How does this happen? With Playwright's .toHaveScreenshot( ) to take a screenshot and the mask method that you want to leave out of the comparisons between the expected and actual screenshots.
- toHaveScreenshot(name): Playwright.dev / Page Assertions
Our Test Site: PracticeSoftwareTesting.com
Butch has a series of tests for the test site Practice Software Testing ( practicesoftwaretesting.com ).
If you go to the site, you can see that there is an ever moving logo, "Toolshop Demo" which has the title "Practice Software Testing - Toolshop" . Let's write a test capturing an image of the page, leaving out the moving logo.
Butch also suggested waiting for everything to be fully loaded with "networkidle". But, according to the Playwright.dev waitForLoadState documentation, it says "Don't use this method for testing, rely on web assertions to assess readiness instead". Hrm. Let's go with that he has for now.
Writing the Visual Test
home.spec.ts
test("visual test", async ({ page }) => {
await page.waitForLoadState("networkidle");
await expect(page).toHaveScreenshot("home-page-no-auth.png", {
mask: [page.getByTitle("Practice Software Testing - Toolshop")],
});
});
The first time you run this test, it fails, since there are not yet any snapshots saved. There is no golden file. The second time you run it, it will pass. If you wanted to bypass this step, and just update the snapshots for this page, you can run. This regenerates any snapshots for the assertions:
- npx playwright test tests/home.spec.ts --update-snapshots
Did a snapshot not match up with the one saved?
You can investigate it in the built-in Playwright reports, showing differences outlined in red, the actual snapshot, the expected snapshots, and see them side-by-side. It also have a feature where you can move slider back and forth horizontally across the image investigating actual and expected snapshots.
Notice that the mask shows that image in an array. Need to mask a bunch of items on the page? You can add them all to the array.
You can also grab screenshots not just of whatever is in the viewport, or the full page beyond the fold, but also compare individual images, or elements such as popup windows.
Visual Comparisons
The ToHaveScreenshot Method
This is all part of Playwright.dev's PageAssertions class, where we are expecting the page for the actual viewable screenshot of the app to match the saved expected viewable screenshot. ( See all Options ). Options you can use:
- fullPage: You can decide to capture the entire scrollable page, setting it as "true".
- mask: You can tell Playwright to ignore certain areas, such as headers or footers if you are just interested in the main area of the site, finding the locators of those areas and mask them. Normally, it will be a pink box in the report to show what is masks, but you can change the maskColor in CSS color format.
- threshold: How strict should colors match? Should it be strict (0) or lax (1) or somewhere in between? The default is 0.2.
- omitBackground: When you want to hide the default white background and capture it as transparent.
- maxDiffPixels & maxDiffPixelRatio: Shows the acceptable amount of pixels that could be different and a ratio of different pixels between 0 and 1. We can configure both using TestConfig.expect. These are turned off by default.
Matching Pixels With Pixelmatch
Playwright uses the pixelmatch JavaScript library ( https://github.com/mapbox/pixelmatch ) which is used to compare actual and expected images.
Configure Visual Checking In Playwright Config
Want to fine tune the above variables? You can update the global configuration in your playwright.config.ts file:
import { defineConfig } from '@playwright/test';
export default defineConfig({
expect: {
toHaveScreenshot: {
// Allow up to 2% of total pixels to change
// (handles layout shifts/dynamic content)
maxDiffPixelRatio: 0.02,
// Increase pixel color tolerance
// (0.2 blends strict pixel match with visual difference)
threshold: 0.2,
// Automatically mask dynamic elements like blinking text cursors
caret: 'hide',
// Ensure CSS animations are frozen on the first frame
animations: 'disabled',
},
},
});
How to Compare Product Cards?
Let's say on the PracticeSoftwareTesting.com site, you wanted to create a visual comparison of the first product card. The code could look something like this:
test('product card matches snapshot', async ({ page }) => {
await page.goto('https://practicesoftwaretesting.com');
// Locate a specific element -- here, the first product card
const productCard = page.locator('[data-test="product"]').first();
// Wait for the image inside the card to fully load before snapping
await productCard.locator('img').waitFor({ state: 'visible' });
// Compare element screenshot against stored baseline
await expect(productCard).toHaveScreenshot('product-card.png');
});
});How To Mask Changing Elements Such As Prices?
Prices for a Shopping Cart will never remain stagnant. They are always changing.
Let's say you just want to do visual checking on the basic layout. Every time the prices change, you don't want the visual test to fail. What you can do is just mask out all the prices, which are with the ids "product-price".
// tests/full-page-screenshot.spec.ts
import { test, expect } from '@playwright/test';
test('homepage layout matches snapshot, prices masked', async ({ page }) => {
await page.goto('https://practicesoftwaretesting.com');
await expect(page).toHaveScreenshot('homepage.png', {
// Mask elements whose content changes between runs
mask: [
page.locator('[data-test="product-price"]'),
],
// Pixel-level tolerance -- useful when anti-aliasing differs
maxDiffPixelRatio: 0.02,
});
});... And if you wanted even more fine tuning, you could choose to Integrate Applitools with Playwright.
Running Visual Testing in CI Platforms
Let's say you are running visual testing in a CI environment: Make sure you generate the snapshots against what is in CI, and not your local environment or there will be pixel differences.
If you are using Playwright's pre-built Docker Image (see docs ) make sure to update the images in the container you are using.
According top TestQuality.com: Playwright Visual Regression Testing: A Production Guide to Baselines, Flake, and CI, "Native Playwright visual regression is free to start and expensive to scale. The cost shows up in CI, not on day one".
The problem is that "Windows, macOS, and Linux render fonts, anti-aliasing, and pixel spacing differently. A baseline captured on a developer's Mac will fail when compared against a screenshot taken on the Linux CI runner, even when the application code has not changed".
The solution? "[A] Dockerfile based on the official Playwright image, plus docker-compose mounting screenshot volumes so developers run the same Linux rendering engine locally that the CI runner uses. A single command brings up the same environment everywhere, which eliminates 'works on my machine' visual discrepancies entirely".
How to manage them? "Playwright visual regression CI integration requires treating visual checks as a separate, isolated pipeline stage. Tag visual tests with @visual, exclude them from standard runs using --grep-invert, and execute them in a dedicated Docker step. This keeps functional feedback fast and quarantines visual flake to a controlled environment".
Any other problems? "Storage is the harder problem. Native baselines live in the Git repository, with one image per browser engine — a ten-test suite covering Chromium, Firefox, and WebKit produces thirty committed images. The repo grows fast, but the bigger pain is approval. GitHub, GitLab, and most version control UIs cannot diff images inline. When a baseline updates, the reviewer sees a binary file change with no visible context. Resolving the conflict requires pulling the branch locally, opening the images side by side, copying the new baseline into the feature branch, and pushing manually. House calls this 'GitHub diff blindness,' and it is the single biggest reason teams move off native after a few months".
No comments:
Post a Comment