Investigating AI: Playwright-Test-Planner and Playwright-Test-Generator

Testing Out Three New Playwright AI Tools:

The other day, I heard about three new Playwright AI tools that came out in October 2025: 

Playwright / Test Agents: "Playwright comes with three Playwright Test Agents out of the box: 🎭 planner, 🎭 generator and 🎭 healer.

"These agents can be used independently, sequentially, or as the chained calls in the agentic loop. Using them sequentially will produce test coverage for your product.

"🎭 planner explores the app and produces a Markdown test plan

"🎭 generator transforms the Markdown plan into the Playwright Test files

"🎭 healer executes the test suite and automatically repairs failing tests"

... So of course, I had to check it out! How well did they do? 

I found it fascinating that Playwright-test-planner could write a detailed test plan, one you could review and edit. And it was amazing that Playwright-test-generator could explore a website and find locators, and organize tests all on its own. 

The problem is that it did make incorrect assumptions when finding locators for common elements such as a "heading". When it logged into a login page, it thought the build in Playwright "heading" would have been enough to find the "Secure Area". Except the heading type was in an "h2" tag, so I had to tweak it a bit.

All in all, it did well, getting me 80% there. If you want your automated test framework to store login credentials, or string messages in a separate data object, or store locators in private variables in page objects, you need to explicitly have prompts stating such. 

AI: Good as a Research Assistant. Bad for Creating GitHub Action Workflows

I've been organizing head-too-head match-ups of various flavors of GitHub Copilot and Cursor, to keep from being bored sending resumes out into the void.  

In the past I have used AI: Code-completion at MassMutual, code review at SELF, and vibe-coding apps for fun during Christmas break. 

The most powerful use I've found with AI? Research Assistant:

• "Here is a list of toolsets. Describe them. Be brief". 
• "What are the release dates of these toolsets?" 
• "Use corporate tech blogs as primary sources".
• "Cite your sources. Provide links". 

New Project: Cursor creates a Playwright + C-Sharp test framework

It's a head-to-head matchup! Cursor AI versus VS Code + GitHub CoPilot battling to create automated test frameworks using MS Playwright + C#. Who creates the best tests? The best GitHub Actions Workflow? The best README docs? And can it be created only using prompts?

In this corner, GitHub CoPilot, with the GitHub project: Login-C-Sharp.

In the other, Cursor AI, with the GitHub project: Cursor-creates-playwright-c-sharp

Let the battle begin!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

New project: Creating an automated test framework in Playwright + C# using GitHub Copilot

When I learned that a company I was interviewing was thinking of pairing Playwright with C#, a computer language I have never worked with before, for an automated test framework for their web application, I was inspired.

The Test

Given a website, such as https://the-internet.herokuapp.com/login, can GitHub Copilot examine the website, and create, through only prompting an automated test framework using C#, NUnit, and Playwright? What if we are using the free version of GPT-4.1?

Presenting a work in progress! 

• Login C Sharp: https://github.com/tjmaher/login-c-sharp  

Want to see what prompts were used for this site? The last section of the README file contains a summary of prompts used to create this project and its documentation, along with the actions Copilot executed. 

Surprisingly, only very few minor manual tweaks of the documentation and code below were needed, such as weird formatting issues in YAML files, and new text in this README placed incorrectly.

WARNING!

Chat-GPT 4.1 has a cutoff date of two years ago. When creating a workflow, GitHub Copilot did not realize that it was using a deprecated version of actions/upload-artifact (v3) causing the workflow to fail. Caveat emptor!

The Results!

So, how did GitHub Copilot + Chat GPT do creating an automation framework? I would say it did so good that it was hideously frustrating when it messed up the simple things.

It's like an eager-to-please junior dev who doesn't completely know the material and doesn't know it isn't reading the latest documentation.

Why would it not know it was implementing out-of-date libraries when creating the GitHub Actions Workflow? It was so sure it had everything correct until I copied-and-pasted the error I received from the GitHub Actions log files and fed the error back to it.

Why does it not read actual documentation? Why does it skip carefully enumerated steps? And why does it always profusely apologize to me while doing the same mistake over and over again?

I feel that it got me 80% there, but it was super frustrating needing to drag it bodily across the finish line.


Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

New Features of Detox Demo: Security Scanning + Android Support + Cross-Platform Builds!

Remember that tiny little two-screen React Native app I created back in December 2025? The one that just had a Login Page and a Secure Area? Well, I may have gone a little overboard adding features to it again.

What started as a simple React Native Login Page demo for my AutomationGuild talk in April 2026 has become... way, way, way too much.

What's new in Detox Demo?

📚 Tools and Technologies Galore!

The project now uses: React Native, Yarn, Detox, Detox CLI, Allure Reports, Snyk, GitHub Actions, GitHub Workflows, GitHub Pages, Metro bundler, CocoaPods, Android Gradle configuration, iPhone simulators and Android emulators, and includes troubleshooting guides for both macOS and Windows.

All open-source. All documented. All completely unnecessary for what is essentially a Login button and a Logout button.

🔐 Snyk Security Scanning

Because even a demo app that has hardcoded credentials (yes, tomsmith and SuperSecretPassword! are right there in plain text in credentials.ts) deserves security scanning!

I've added a new security.yml GitHub Actions workflow that:

• Scans package.json and yarn.lock for vulnerable npm packages
• Runs Static Application Security Testing (SAST) on the source code
• Uploads results to GitHub Code Scanning so they appear in the repository's Security tab

It runs on every push to main, every pull request, and you can kick it off manually. 

Snyk is free for public repositories. If it's free, it's for me, I'll take three. 

🤖 Android Support

The app now runs on Android! I've added:

• android-regression.yml - A full CI/CD pipeline for Android emulator testing
• Cross-platform Gradle commands in .detoxrc.js that automatically detect Windows vs macOS
• Complete Setup for Android documentation in the README

Run locally on Windows 11 or macOS:

yarn start          # Start Metro in one terminal
yarn detox:android  # Build and test in another

All 5 tests pass:

• ✅ Secure Area Flow: 2 tests
• ✅ Login Flow: 3 tests

🪟 Windows 11 Local Development

Since I'm developing on a Windows 11 machine these days, I asked GitHub CoPilot to generate comprehensive Setup for Windows 11 Local Development guide covering:

• Android SDK installation
• AVD creation
• Environment variable setup
• Troubleshooting common issues

Plus a matching Setup for macOS Local Development guide for MacBook users.

🧹 GitHub Copilot Code Review Fixes

I now run GitHub Copilot's code review feature on the codebase. All the source files created by GitHub Copilot now have a "Created by GitHub Copilot" comment at the top, because credit where credit is due!

And thank you, GitHub Copilot for the rough draft of this post, for copying my stream-of-consciousness writing style, and the following suggestion: 

What's the most over-engineered demo project YOU'VE ever built? Leave some notes in the comments below! 👇


Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Creating a GitHub Actions Workflow for Android Detox Testing with GitHub CoPilot? What Could Go Wrong?

Last month, I shared my experience using GitHub Copilot to create a React Native app from scratch to be used in my DetoxDemo project in my article, First Time Using GitHub CoPilot to Create a ReactNative LoginPage app. What Could Go Wrong?

This time, I used GitHub Copilot (Claude Opus 4.5) to create a GitHub Actions CI/CD workflow for running Detox end-to-end tests on Android. While GitHub CoPilot is incredibly powerful, it still required significant human guidance to get the workflow passing.

Detox Demo: https://github.com/tjmaher/detox-demo

I had a working GitHub Actions Workflow with ios-regression.yml and asked Copilot to create an Android version that matched. Despite this instruction, I had to repeatedly ask Copilot to compare against the iOS workflow to create the Android workflow, android-regression.yml.

The result? 14 commits, 17 hours, and a lot of lessons learned. Here's the timeline of what went wrong, and what finally worked:

The Facebook Ecosystem: React, React Native, Metro, and Yarn

Whenever attempting to construct a new automation framework from scratch, it can be difficult figuring out which automated testing toolsets should be used. This is why, before I do anything, I research the new tools and technologies used to create the app I will be testing, hoping to see if there are any industry standards already out there. I’ve paired Angular with Protractor, Ruby with Watir and Capybara. What should I pair with a React Native mobile app? Appium, like I did with the Stop & Shop mobile apps? Or is there something else?
Before building an automated testing framework, I had to do some research on the toolsets in the Facebook ecosystem that SELF’s mobile app used: React, React Native, Metro, and Yarn.

GitHub:

• https://github.com/facebook/react
• https://github.com/facebook/react-native
• https://github.com/facebook/metrohttps://github.com/yarnpkg/berry