Testing Out Three New Playwright AI Tools:
The other day, I heard about three new Playwright AI tools that came out in October 2025:
Playwright / Test Agents: "Playwright comes with three Playwright Test Agents out of the box: 🎠planner, 🎠generator and 🎠healer.
"These agents can be used independently, sequentially, or as the chained calls in the agentic loop. Using them sequentially will produce test coverage for your product.
"🎠planner explores the app and produces a Markdown test plan
"🎠generator transforms the Markdown plan into the Playwright Test files
"🎠healer executes the test suite and automatically repairs failing tests"
... So of course, I had to check it out! How well did they do?
I found it fascinating that Playwright-test-planner could write a detailed test plan, one you could review and edit. And it was amazing that Playwright-test-generator could explore a website and find locators, and organize tests all on its own.
The problem is that it did make incorrect assumptions when finding locators for common elements such as a "heading". When it logged into a login page, it thought the build in Playwright "heading" would have been enough to find the "Secure Area". Except the heading type was in an "h2" tag, so I had to tweak it a bit.
All in all, it did well, getting me 80% there. If you want your automated test framework to store login credentials, or string messages in a separate data object, or store locators in private variables in page objects, you need to explicitly have prompts stating such.
Out of the box, you aren't going to get the best solution. You still need a Senior SDET to guide the AI in building the automation framework.
After all my experimentation with various coding AIs these past several months, I don't think they can replace SDETs, but they will save on a lot of typing!
Sidenote: Thank you Butch Mayhew!
If you ever really enjoy a course on LinkedIn Learning, reach out to the author and let them know. Maybe you might strike up a wonderful conversation! Butch Mayhew, Playwright Ambassador, referred me to two tools that came out a few months ago, in October of 2025: Playwright-Test-Planner and Playwright-Test-Generator, two Playwright test agents that you can plug into GitHub Copilot.
Need expert Playwright services? Butch has set up a shingle with Howdy QA: "Save time and reduce testing costs with our expert Playwright automation services. From migration to training, we help you build a quality testing culture. From Zero to Automated with Playwright, Test Automation Health Audit, API Test Automation and Training & Education".
Other Playwright resources from Butch:
- Book: Hands-On Automated Testing with Playwright: Create fast, reliable, and scalable tests for modern web apps with Microsoft's automation framework
- LinkedIn Learning Courses: Learning Playwright and Playwright Essential Training: Abstractions, Fixtures, and Complex Scenarios
- Blog: https://playwrightsolutions.com/
Setting Up Windows 11:
First, I created a new directory on my Windows 11 desktop at home. I tend to store source code of my projects in D:\src. With VS Code, I opened up a new Power Shell terminal, changed to the D:\src directory and made a new directory:
- cd D:\src
- mkdir playwright-generate-plan
... then in VS Code I did File -> Open Folder and opened up the playwright-generate-plan directory I just created in D:\spec\playwright-generate-plan
Installing Playwright:
According to Playwright Docs / Intro all I had to do in the terminal was to:
- npm init playwright@latest
- I selected the defaults of using TypeScript, place the end-to-end tests in "tests", and "Y" to create a GitHub Actions workflow, so I could create an eventual CI/CD pipeline.
- I chose it to install Chromium, Firefox, and Webkit (emulating Mac Safari) as the internal browsers.
After I installed it all, it printed out some helpful commands:
npx playwright test
Runs the end-to-end tests.
npx playwright test --ui
Starts the interactive UI mode.
npx playwright test --project=chromium
Runs the tests only on Desktop Chrome.
npx playwright test example
Runs the tests in a specific file.
npx playwright test --debug
Runs the tests in debug mode.
npx playwright codegen
Auto generate tests with Codegen.
We suggest that you begin by typing:
npx playwright test
And check out the following files:
- .\tests\example.spec.ts - Example end-to-end test
- .\playwright.config.ts - Playwright Test configuration
Visit https://playwright.dev/docs/intro for more information. Next, I wanted to check out Playwright.dev / Docs / Agents which mentioned to run:
- npx playwright init-agents --loop=vscode
npx playwright test
Runs the end-to-end tests.
npx playwright test --ui
Starts the interactive UI mode.
🎠Using project "chromium" as a primary project
🌱 tests\seed.spec.ts - default environment seed file
🤖 .github\agents\playwright-test-generator.agent.md - agent definition
🤖 .github\agents\playwright-test-healer.agent.md - agent definition
🤖 .github\agents\playwright-test-planner.agent.md - agent definition
🔧 .vscode\mcp.json - mcp configuration
🔧 .github\workflows\copilot-setup-steps.yml - GitHub Copilot setup steps
🔧 TODO: GitHub > Settings > Copilot > Coding agent > MCP configurationOpen GitHub Copilot and plan away!
Opening up a new GitHub Copilot session, I selected from the dropdowns:
- Local
- playwright-test-planner
- GPT-4.1 (the free version that would not use up any tokens)
... then I gave it the prompt...
Using Playwright and playwright-test-planner, outline positive and negative test cases for https://the-internet.herokuapp.com/login
* Go to https://the-internet.herokuapp.com/login and check that the title is "Login Page".
* Login as tomsmith / SuperSecretPassword!
* Once on Secure Area, select the Logout button and make sure that when back on the Login Page, that "You logged out of the secure area!" appears. Then, playwright-test (MCP server) went to work!
Instead of having to approve every few seconds, I set it to "Always Allow" without needing to review.
After a minute or two it came up with "login.plan.md". Not bad!
The Testplan is Auto-generated!
---
# Login Page Test Plan
## Application Overview
Test plan for the login functionality at https://the-internet.herokuapp.com/login, covering positive and negative scenarios including login, logout, and error handling.
## Test Scenarios
### 1. Login and Logout Functionality
**Seed:** `tests/seed.spec.ts`
#### 1.1. Positive Login and Logout Flow
**File:** `tests/login-positive.spec.ts`
**Steps:**
1. Navigate to https://the-internet.herokuapp.com/login
- expect: The page loads successfully.
2. Verify the visible heading is 'Login Page'
- expect: The heading 'Login Page' is present.
3. Enter 'tomsmith' in the Username field
- expect: Username field contains 'tomsmith'.
4. Enter 'SuperSecretPassword!' in the Password field
- expect: Password field contains 'SuperSecretPassword!'.
5. Click the Login button
- expect: Redirected to the Secure Area page.
- expect: A message 'You logged into a secure area!' is displayed.
6. Click the Logout button
- expect: Redirected back to the Login Page.
- expect: A message 'You logged out of the secure area!' is displayed.
#### 1.2. Negative Login Scenarios
**File:** `tests/login-negative.spec.ts`
**Steps:**
1. Navigate to https://the-internet.herokuapp.com/login
- expect: The page loads successfully.
2. Enter invalid username 'wronguser' and invalid password 'wrongpass'
- expect: An error message 'Your username is invalid!' is displayed.
3. Leave both Username and Password fields blank and click Login
- expect: An error message 'Your username is invalid!' is displayed.
Generate the Tests:
Next, I switched the dropdown to "playwright-test-generator", and prompted it with:
"Using Playwright + TypeScript and playwright-test-generator, generate the tests for specs/login.plan.md. Refactor locators into page objects, and common elements into a base page".
It then came back with:
"To implement your request, I will generate Playwright tests for both positive and negative login scenarios, refactoring locators into page objects and extracting common elements into a base page. No page object files currently exist, so I will create:
* BasePage (for shared elements/methods)
* LoginPage (for login form and actions)
* SecureAreaPage (for secure area and logout)
I will then generate the tests using these page objects.It automatically generated page objects for base.page.ts,login.page.ts,and secure-area.page.ts in the tests/pages folder.
Minor Error Locating Secure Area Heading
When I went to run the tests, though, the negative tests passed, but the "login-positive.spec.ts" tests failed, but since there were no screenshots, it was difficult to debug what was happening, so I asked GitHub Copilot to have playwright.config.ts turn video and screenshots to ON.
It did have one small problem: It couldn't find the label "Secure Area"..
- this.heading = page.getByRole('heading', { name: 'Secure Area' });
Examining the page, it showed that the heading was actually an "h2" tag. Once I update it to
- this.heading = page.locator('h2')
... everything worked out fine!
What a quick way to stand up a simple project. Thank you, Butch for referring me to the tools!
-T.J. Maher
Software Engineer in Test
BlueSky | YouTube | LinkedIn | Articles
No comments:
Post a Comment