Investigating AI: Playwright-Test-Planner and Playwright-Test-Generator

Testing Out Three New Playwright AI Tools:

The other day, I heard about three new Playwright AI tools that came out in October 2025: 

Playwright / Test Agents: "Playwright comes with three Playwright Test Agents out of the box: 🎭 planner, 🎭 generator and 🎭 healer.

"These agents can be used independently, sequentially, or as the chained calls in the agentic loop. Using them sequentially will produce test coverage for your product.

"🎭 planner explores the app and produces a Markdown test plan

"🎭 generator transforms the Markdown plan into the Playwright Test files

"🎭 healer executes the test suite and automatically repairs failing tests"

... So of course, I had to check it out! How well did they do? 

I found it fascinating that Playwright-test-planner could write a detailed test plan, one you could review and edit. And it was amazing that Playwright-test-generator could explore a website and find locators, and organize tests all on its own. 

The problem is that it did make incorrect assumptions when finding locators for common elements such as a "heading". When it logged into a login page, it thought the build in Playwright "heading" would have been enough to find the "Secure Area". Except the heading type was in an "h2" tag, so I had to tweak it a bit.

All in all, it did well, getting me 80% there. If you want your automated test framework to store login credentials, or string messages in a separate data object, or store locators in private variables in page objects, you need to explicitly have prompts stating such. 

Out of the box, you aren't going to get the best solution. You still need a Senior SDET to guide the AI in building the automation framework. 

After all my experimentation with various coding AIs these past several months, I don't think they can replace SDETs, but they will save on a lot of typing! 

Sidenote: Thank you Butch Mayhew!

If you ever really enjoy a course on LinkedIn Learning, reach out to the author and let them know. Maybe you might strike up a wonderful conversation! Butch Mayhew, Playwright Ambassador, referred me to two tools that came out a few months ago, in October of 2025: Playwright-Test-Planner and Playwright-Test-Generator, two Playwright test agents that you can plug into GitHub Copilot. 

• View Code: https://github.com/tjmaher/playwright-generate-plan

Need expert Playwright services? Butch has set up a shingle with Howdy QA: "Save time and reduce testing costs with our expert Playwright automation services. From migration to training, we help you build a quality testing culture. From Zero to Automated with Playwright, Test Automation Health Audit, API Test Automation and Training & Education". 

Other Playwright resources from Butch:

• Book: Hands-On Automated Testing with Playwright: Create fast, reliable, and scalable tests for modern web apps with Microsoft's automation framework 
• LinkedIn Learning Courses: Learning Playwright  and Playwright Essential Training: Abstractions, Fixtures, and Complex Scenarios 
• Blog: https://playwrightsolutions.com/

AI: Good as a Research Assistant. Bad for Creating GitHub Action Workflows

I've been organizing head-too-head match-ups of various flavors of GitHub Copilot and Cursor, to keep from being bored sending resumes out into the void.  

In the past I have used AI: Code-completion at MassMutual, code review at SELF, and vibe-coding apps for fun during Christmas break. 

The most powerful use I've found with AI? Research Assistant:

• "Here is a list of toolsets. Describe them. Be brief". 
• "What are the release dates of these toolsets?" 
• "Use corporate tech blogs as primary sources".
• "Cite your sources. Provide links". 

These prompts provide excellent documentation that allows me to deepen my own toolset education. 

When I ask AI to build this study guide, AI is teaching me how I can teach myself. 

When I ask AI to do something for me, the only thing its doing is teaching me is how to craft a prompt.

See, when AI does eventually screw up, the only thing I can do to "fix" it is try to craft another prompt, hoping that maybe this time it can see and correct its mistake. And if it can't... ask it again? And again? Tenth time is the charm?

Where I find AI fouling up time and time again? Creating CI/CD pipelines using GitHub Actions Workflows running automated tests in Android emulators. 

• It doesn't realize that GitHub actions have new versions that have been released in the past year.
• It declares you should be using a Mac runner since it is more stable. No, a Linux runner is better! No, a Mac runner! It flip flops on them between code reviews. 
• If you ask it to shift to a build / test / report stage, it always forgets to upload the artifacts in one stage so they can be downloaded to another. 
• It suggests Intel-based Android emulators to be used on Mac OS runners, only recognizing how it fowled up if you copy-and-paste the error to it. 
• It will erase and change the comments you placed in the workflow if you do not watch out. 
• It keeps wanting to go out of the box you placed it in, and rearrange disorganized code that runs elsewhere in the project, and change it to pretty code that has hidden errors in it. 

Why do they get iOS workflows mostly right and Android workflows mostly wrong? Who knows! 

All I can say is that if you are going to add AI into your workflow, learn some breathing exercises. You will need them to work through the frustration you are about to face.

Want to see other projects where I have used AI? Check out the list of programming projects on my blog, where I have sample code I have written for the past ten years at https://www.tjmaher.com/p/programming-projects.html 

What do you find AI constantly screws up? How do you fix it? Leave comments below! 

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

New Project: Cursor creates a Playwright + C-Sharp test framework

It's a head-to-head matchup! Cursor AI versus VS Code + GitHub CoPilot battling to create automated test frameworks using MS Playwright + C#. Who creates the best tests? The best GitHub Actions Workflow? The best README docs? And can it be created only using prompts?

In this corner, GitHub CoPilot, with the GitHub project: Login-C-Sharp.

In the other, Cursor AI, with the GitHub project: Cursor-creates-playwright-c-sharp

Let the battle begin!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

New project: Creating an automated test framework in Playwright + C# using GitHub Copilot

When I learned that a company I was interviewing was thinking of pairing Playwright with C#, a computer language I have never worked with before, for an automated test framework for their web application, I was inspired.

The Test

Given a website, such as https://the-internet.herokuapp.com/login, can GitHub Copilot examine the website, and create, through only prompting an automated test framework using C#, NUnit, and Playwright? What if we are using the free version of GPT-4.1?

Presenting a work in progress! 

• Login C Sharp: https://github.com/tjmaher/login-c-sharp  

Want to see what prompts were used for this site? The last section of the README file contains a summary of prompts used to create this project and its documentation, along with the actions Copilot executed. 

Surprisingly, only very few minor manual tweaks of the documentation and code below were needed, such as weird formatting issues in YAML files, and new text in this README placed incorrectly.

WARNING!

Chat-GPT 4.1 has a cutoff date of two years ago. When creating a workflow, GitHub Copilot did not realize that it was using a deprecated version of actions/upload-artifact (v3) causing the workflow to fail. Caveat emptor!

The Results!

So, how did GitHub Copilot + Chat GPT do creating an automation framework? I would say it did so good that it was hideously frustrating when it messed up the simple things.

It's like an eager-to-please junior dev who doesn't completely know the material and doesn't know it isn't reading the latest documentation.

Why would it not know it was implementing out-of-date libraries when creating the GitHub Actions Workflow? It was so sure it had everything correct until I copied-and-pasted the error I received from the GitHub Actions log files and fed the error back to it.

Why does it not read actual documentation? Why does it skip carefully enumerated steps? And why does it always profusely apologize to me while doing the same mistake over and over again?

I feel that it got me 80% there, but it was super frustrating needing to drag it bodily across the finish line.


Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

New Features of Detox Demo: Security Scanning + Android Support + Cross-Platform Builds!

Remember that tiny little two-screen React Native app I created back in December 2025? The one that just had a Login Page and a Secure Area? Well, I may have gone a little overboard adding features to it again.

What started as a simple React Native Login Page demo for my AutomationGuild talk in April 2026 has become... way, way, way too much.

What's new in Detox Demo?

📚 Tools and Technologies Galore!

The project now uses: React Native, Yarn, Detox, Detox CLI, Allure Reports, Snyk, GitHub Actions, GitHub Workflows, GitHub Pages, Metro bundler, CocoaPods, Android Gradle configuration, iPhone simulators and Android emulators, and includes troubleshooting guides for both macOS and Windows.

All open-source. All documented. All completely unnecessary for what is essentially a Login button and a Logout button.

🔐 Snyk Security Scanning

Because even a demo app that has hardcoded credentials (yes, tomsmith and SuperSecretPassword! are right there in plain text in credentials.ts) deserves security scanning!

I've added a new security.yml GitHub Actions workflow that:

• Scans package.json and yarn.lock for vulnerable npm packages
• Runs Static Application Security Testing (SAST) on the source code
• Uploads results to GitHub Code Scanning so they appear in the repository's Security tab

It runs on every push to main, every pull request, and you can kick it off manually. 

Snyk is free for public repositories. If it's free, it's for me, I'll take three. 

🤖 Android Support

The app now runs on Android! I've added:

• android-regression.yml - A full CI/CD pipeline for Android emulator testing
• Cross-platform Gradle commands in .detoxrc.js that automatically detect Windows vs macOS
• Complete Setup for Android documentation in the README

Run locally on Windows 11 or macOS:

yarn start          # Start Metro in one terminal
yarn detox:android  # Build and test in another

All 5 tests pass:

• ✅ Secure Area Flow: 2 tests
• ✅ Login Flow: 3 tests

🪟 Windows 11 Local Development

Since I'm developing on a Windows 11 machine these days, I asked GitHub CoPilot to generate comprehensive Setup for Windows 11 Local Development guide covering:

• Android SDK installation
• AVD creation
• Environment variable setup
• Troubleshooting common issues

Plus a matching Setup for macOS Local Development guide for MacBook users.

🧹 GitHub Copilot Code Review Fixes

I now run GitHub Copilot's code review feature on the codebase. All the source files created by GitHub Copilot now have a "Created by GitHub Copilot" comment at the top, because credit where credit is due!

And thank you, GitHub Copilot for the rough draft of this post, for copying my stream-of-consciousness writing style, and the following suggestion: 

What's the most over-engineered demo project YOU'VE ever built? Leave some notes in the comments below! 👇


Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Creating a GitHub Actions Workflow for Android Detox Testing with GitHub CoPilot? What Could Go Wrong?

Last month, I shared my experience using GitHub Copilot to create a React Native app from scratch to be used in my DetoxDemo project in my article, First Time Using GitHub CoPilot to Create a ReactNative LoginPage app. What Could Go Wrong?

This time, I used GitHub Copilot (Claude Opus 4.5) to create a GitHub Actions CI/CD workflow for running Detox end-to-end tests on Android. While GitHub CoPilot is incredibly powerful, it still required significant human guidance to get the workflow passing.

Detox Demo: https://github.com/tjmaher/detox-demo

I had a working GitHub Actions Workflow with ios-regression.yml and asked Copilot to create an Android version that matched. Despite this instruction, I had to repeatedly ask Copilot to compare against the iOS workflow to create the Android workflow, android-regression.yml.

The result? 14 commits, 17 hours, and a lot of lessons learned. Here's the timeline of what went wrong, and what finally worked:

[ View the Pull Request ]

The Stats

Total Commits: 14 commits

Time Span: ~17 hours

• Started: Feb 3, 2026 at 9:54 PM EST
• Finally Passed: Feb 4, 2026 at 3:15 PM EST

The Facebook Ecosystem: React, React Native, Metro, and Yarn

Whenever attempting to construct a new automation framework from scratch, it can be difficult figuring out which automated testing toolsets should be used. This is why, before I do anything, I research the new tools and technologies used to create the app I will be testing, hoping to see if there are any industry standards already out there. I’ve paired Angular with Protractor, Ruby with Watir and Capybara. What should I pair with a React Native mobile app? Appium, like I did with the Stop & Shop mobile apps? Or is there something else?
Before building an automated testing framework, I had to do some research on the toolsets in the Facebook ecosystem that SELF’s mobile app used: React, React Native, Metro, and Yarn.

GitHub:

• https://github.com/facebook/react
• https://github.com/facebook/react-native
• https://github.com/facebook/metrohttps://github.com/yarnpkg/berry

Hands on Automated Testing with Playwright is the start of a wonderful conversation with the Playwright community!

I really loved Faraz Kelhini & Butch Mayhew's new book, "Hands on Automated Testing with Playwright: Create fast, reliable, and scalable tests for modern web apps with Microsoft's automation framework". 

Butch Mayhew is a Playwright Ambassador, dedicated to helping others, and it shows! I fully recommend this book -- and Butch's many LinkedIn Learning Playwright courses -- for those attempting to understand Playwright. 

The real beauty of the book is that it feels like only the start of a continuing conversation: 

• Sample code is included: Just like Butch's courses, it provides a GitHub repo chock-full of code examples where Butch and Faraz walks through the examples chapter by chapter so the reader can see implemented the concepts that they both explain. 
• Reference links is included: Need to do a deep dive on a topic? The authors have included links to the primary sources, such as Faraz Kelhini's article, Understanding Shadow Dom. (2019)
• QR codes that connect to the Playwright community: Want to connect with the Playwright community at large? See a Playwright community calendar? Scan the included QR codes in the books.

It's no problem if you have never used Playwright before. Readers are walked through installing the toolsets, writing & running their first tests, setting up VS Code, and how to configure Playwright settings. 

The book also walks the reader through chapters on AI-Powered Test Generation using GitHub CoPilot and the Playwright Model Context Protocol (MCP), generating tests with Playwright's Codegen feature. 

Thank you so much for the advance copy, Butch! 

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles
GitHub repo

#OpenToWork: Looking for My Next Test Automation Adventure!

Hey everyone! I'm looking for my next adventure in test automation.

After an incredible but way too short run at SELF ID, building mobile test frameworks for our React Native app, I'm ready to bring my expertise to a new team that values quality, automation, and collaboration.

What I bring to the table:

With a decade of experience as an automation developer, I don't just write tests. Embedded with a development team, I learn about the wants and needs of the stakeholders - the developers, the designers, the business analysts, the business itself - and construct a test automation framework, two week sprint by two week sprint, that truly fits their needs. [ See my Programming Projects ]

The quicker the automation framework is stood up, the quicker I can get to the truly fun stuff: Making sure the brand-new untested features fresh off the developer's local machine meets not just the spec, wireframes, requirements, and design, but also to make sure it matches those undocumented expectations that were discussed but may have not been carefully documented... something that AI will never be able to do.

My most recent project? DetoxDemo (https://github.com/tjmaher/detox-demo) - a complete mobile automation framework showcasing:

• Mobile test automation with Detox + TypeScript for React Native iOS apps
• Page Object architecture that keeps tests clean and maintainable
• Allure Reports integration with visual test results published via CI/CD
• GitHub Actions workflows with configurable test execution options

On top of my extensive software testing experience, the last ten years has been focused on test automation. I've built automation frameworks from scratch at SELF Id and ThreatStack. I’ve created automated development courses for Test Automation University. I’ve written extensively about testing (check out my blog "Adventures in Automation"), and organized the Ministry of Testing - Boston meetup for years.

What I'm looking for:

Software Engineer in Test or SDET roles where I can design automation strategies, sharing my testing worldview with developers, and build frameworks that the entire team can use to check their work before merging into main. Remote work is preferred, but I am open to hybrid opportunities in the Boston area or Southeastern Massachusetts.

My toolbox includes: Detox, some Playwright, Selenium WebDriver, Ruby/Capybara, Java, TypeScript/JavaScript, React Native, CI/CD pipelines, and Allure Reports. But more importantly - I know how to research new tools, validate stakeholder needs, and implement solutions that fit your tech stack.

If you know of opportunities, I'd love to connect. Drop me a message or comment below. I'm always happy to chat about testing, automation, or that one flaky test that's driving you crazy.


I will be speaking at Joe Colantonio's TestGuild in April 2026. See you then!

#OpenToWork #SDET #TestAutomation #SoftwareEngineering #QualityEngineering


Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

AI wants to take over QA? Let it!

Someone on LinkedIn posted: "The smartest engineering leaders I know have been divesting from manual QA for years now and I think that bet is going to pay off big in today's world.

"If your development process still relies heavily on manual QA and your engineers are now using AI-assisted coding tools, you've created a new bottleneck".

My response? Let it! 

I am all in favor of outsourcing the "boring stuff" to automated tests or AI. Who really wants to check for the umpteenth time that the same page in the web app has the correct working functionality on Chrome, Firefox, MS Edge, Mac Safari, and all the various screen sizes. BORING!

Now, it takes a real software tester to make sure the brand-new untested features fresh off the developer's local machine meets not just the spec, wireframes, requirements, and design, but also to make sure it matches those undocumented expectations that were discussed but may have not been carefully documented. Or that the user experiences matches how the designer really wanted the web app to operate. Or that the user experience doesn't change too much when you operate the web app in the wild.

That all is fun part of the job ... realizing during testing that the business requirement or the design were actually unclear, and that you as a tester found a unique edge case.

Let AI do the mindless drudgery. Just as long as I can still focus on the fun stuff! 

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

DetoxDemo: Now with more GitHub Action Workflow CI/CD Options!

Have you ever known that you probably should have gone to bed hours ago, but you were doing something so fun, you didn't want to stop? I am like that with my toy React Native application, DetoxDemo, which I created as part of my presentation to the AutomationGuild in April 2026.

• DetoxDemo: https://github.com/tjmaher/detox-demo/ 

Late last night, after pouring over Wix's Detox Docs for Artifacts, I decided I wanted to implement that in my GitHub Actions CI/ CD Workflow. 

Want to kick off a job to run all the Login tests in the CI/ CD platform using the GitHub Actions workflow? With the DetoxDemo GitHub:

• Go to Actions -> View all Workflows
• Under the Actions column to the left, select Build & Test iOS
• Select the [Run workflow] button to see all the choices I set up in the ios-regression.yml configuration file under the on: workflow_dispatch -> inputs
• Say you were a developer that wanted to test out their JIRA-123 branch code before merging, under "Use workflow from" they could choose branch JIRA-123 here instead of running against the main branch.
• Which test suite would you like to run? Login? SecureArea? Default is "all".
• Which iPhone 16 would you like to run the tests on? Regular iPhone 16, Pro, or Pro Max? Or maybe an iPad Mini, Air, or Pro?
• What log level? Select any range from the very verbose "trace", to throwing alerts only if things are "fatal". Default is "info".
• What level of artifacts do you want to capture for logs, screenshots, or videos? All, just failing, or none?
• Do you want to run performance testing with Detox Instruments? We have that option! Still looking how the Wix Incubator's Detox Instruments works with CI/CD.
• Or you can just scroll down to the bottom and select [Run Workflow] and kick off the default values set up in ios-regression.yml
• A new "Build & Test iOS" run will be created. Feel free to click into the run to see it run through the build -> test -> publish-allure-reports -> cleanup stages where you can see all Homebrew, RubyGems, Cocoapods, Node.js, and Applesimutils are configured and run.
• If you click into the "build" stage, you can see it work through tasks such as "Set up job", "Checkout repository", "Setup Homebrew", "Setup Ruby", "Cache Homebrew and RubyGems", etc. It takes 30 minutes for a Detox-embedded build to be generated.

New Features of Detox Demo: CI/CD + Logging + Reports, oh my!

To prepare for the talk I'll be giving to the TestGuild in a few months, since I can't share actual source code of the SELF Mobile app or the automation source code I've written on-the-job, I created a toy project, Detox Demo at https://github.com/tjmaher/detox-demo, a React Native app tested with Wix's Detox + TypeScript. 

It's got reports using Detox's Allure Adapter

It's got GitHub Pages integration at https://tjmaher.github.io/detox-demo/ios/ ! 

It's got logs you can examine! 

It's got CI / CD with GitHub Action workflows, which you can kick off manually by branch, by cron job, or by each time you push code into main! 

And it's got documentation up the wazoo in the README file, since I just don't know how to be brief.

And it's all open-source, so you can examine the source code to see how things were set up. 

DetoxDemo is a working React Native demo app that gives examples of:

• Mobile automation tests written in Detox + TypeScript
• Commonly used test code refactored into Page Objects
• Common methods used by Page Objects refactored into a Base Page
• Log reports published after a test run, written like a manual test plan testers can follow
• Pre-written yarn scripts to build and test stored in a package.json
• detox-allure2-adapter set up in .detoxrc.js and the e2e/jest.config.js
• Allure Reports configured to show historical data
• CI/CD provided by GitHub Action workflow
• A working React Native mobile app for iOS complete with source code
• A detailed README documenting the Project Structure, and all the setup for the tools and technologies of this project, along with listing various historical tidbits.
• Scalable Vector Graphic (SVG) showing the DetoxDemo iPhone desktop icon and a setup script generating various sizes of icons.

My LinkedIn Year in Review - AI Style head to head matchup between Coauthor.Studio and Claude.ai.

Thank you, Vernon Richards for referencing Rewind.coauthor.studio ... I love the cute LinkedIn Rewind graphic it produced! I think it was worth the 99 cents.

The AI generated post it produced. though, it just didn't sound like me. Too long. Not chatty or informal enough. The phrasing was off as if instead of bullet points about my year it was trying to unfold a grand epic story that just wasn't there. Then, I then asked Claude.ai to edit the post it so it copied my writing style from the Media section of this blog ... and it was scary how similar to me it sounded!

I'll copy-and-paste the CoAuthor.studio and Claude.ai and put then in the post so people can compare the language.

Hey, a software tester's gotta test!

LinkedIn Profile: https://www.linkedin.com/in/tjmaher1/

First Time Using GitHub CoPilot to Create a ReactNative LoginPage app. What Could Go Wrong?

Do you want to practice mobile test automation development but need a React Mobile app to test against? I have one for you: DetoxDemo, an open-source app with a Login screen, built with ReactNative + TypeScript + Mobile automation written in Detox.

The automated tests I coded myself, but the app is all GitHub CoPilot. This article is about how CoPilot stumbled, fell flat on its face, stumbled, fell again, took a breather, but eventually, after many failed attempts, limped across the finish line, creating exactly the React Native app I had been looking for.

 

Ten years ago, back when I was a newbie automation developer, I discovered a test site created by Dave Haefner: The-Internet, a website where newbie automation developers could practice their craft. Dave Haefner, the author of the Elemental Selenium newsletter and the Selenium Guidebook, two resources that really helped me out early in my career, created the site to help the software testing community.

To practice what I was learning on-the-job at my first automation development position putting together Selenium WebDriver + Java automation framework, I created, testing against that site, a toy project, Testing The-Internet and started blogging about it. I gave imaginary readers code walkthroughs how one could write a framework to test The-Internet’s Login Page. The act of being forced to explain what I am doing in various programming projects and why to imaginary readers has helped me in many parts of my career, whether it is playing around with REST Assured, setting up an Appium framework, or creating an automation framework with Ruby + Capybara.

Ten years later, to practice what I am learning at SELF ID, Inc. putting together an automation framework for their React Native mobile application using Wix’s Detox + TypeScript, I wanted to put together another toy project. Call it DetoxDemo. The problem was that, although I knew what the automation part would look like, I didn’t know of a React Native mobile application to test against. Wishing there was a React Native app like Dave Haefner’s The-Internet, I had a crazy idea:

Why not try to vibe-code a React Native mobile app, creating one with GitHub CoPilot, that emulates his The-Internet Login page?

Sure, there were some minor hurdles, the first being that I don’t know how to “vibe-code”. I’ve been using GitHub CoPilot for a year at two different workplaces, but nothing to this extent.

The second hurdle is that I am not a software developer. With React Native applications, I have been a manual and an automated tester. I just don’t know React Native development.

Creating an entire app using a toolset I am unfamiliar with in a language I don’t understand? What could go wrong?

DetoxDemo, my first vibe-coded React Native app, is now complete

For the past ten years, I've been blogging about writing test automation on this blog, writing toy projects to help practice what I am doing on the job, writing articles and courses about test automation, and storing coding projects at https://github.com/tjmaher.

Starting in January, my next project will be blogging and walking readers through my experience putting together a mobile automation framework at SELF ID, testing their SELF ID React Native mobile app where users can create, store, and share their digital identity. I'm doing all of this in preparation for my upcoming talk to Joe Colantonio's Automation Guild this coming Spring. 

At SELF, I've been using Detox, a grey-box automation framework created by Wix to test Wix's in-house mobile application that the company generously open-sourced. 

Since I am planning to start blogging about putting together a mobile test automation framework in Detox I needed an app to test against... so I vibe-coded an app! Presenting: DetoxDemo!

• Detox Demo: https://github.com/tjmaher/detox-demo

My first vibe-coded React Native application is now complete! 

I based the app on Dave Haefner's The - Internet / Login, a site I used when I was teaching myself automation development writing Selenium + Java tests back in July 2015 in my first project "Testing The-Internet".

Beware the Hallucinations When Coding with AI

Make sure to take what AI gives you with a grain of salt. It can help point you in the right direction as a first pass, but be wary. It might not really understand your question or the material.

When writing a UI automated test to confirm that a user could not continue using our React Native application without first accepting the Terms & Conditions, in order to see if a button was initially enabled or not, GitHub CoPilot suggested to use a Detox method ".toBeDisabled()". The problem? After delving into the Detox documentation and source code I found out this method doesn't actually exist.

When I told the AI it did not exist, it apologized profusely, then suggested I erase that bit of code it wrote, along with the code of other unrelated tests I was working on. When I finally found a solution, and ran it by GitHub CoPilot, after reviewing it, the AI wanted to replace the code with the non-existent method again.

... Eh, it's slightly better than frantically searching through Stack Overflow's archive and interpreting the solutions you find there.

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Two months at SELF already? Happy Anniversary!

I am happy to announce that I have found a new job at SELF ID, a start-up where I am writing a test automation suite for their Android and iPhone application. In the past two months I've been building an automated test suite, checking that you can create a new SELF identity, Sign in with your SELF to a demo site, and store the identity you created on the blockchain.  

Testing tools I am using:

• React Native app testing: Detox + TypeScript. Detox is more grey-box than black box automation, since you embed it in your React Native application so it can monitor the internals of the app, making sure that pages are loaded, network calls are finished, and animations are complete, reducing flakiness. (Detox site: How it Works).
• Web Application testing: Playwright + TypeScript to test our demo sites showcasing how users can "Sign In With Your SELF". Who would have thought a decade ago that Microsoft would be all-in with the open source? Microsoft Playwright, Microsoft's version of JavaScript. Pair it with Microsoft's VB Code + Microsoft's GitHub CoPilot we have powered by Claude.ai. 
• API Testing: I am in the middle of researching Pact.io to do contact testing for our Multi-Dimensional Identity Protocol (MDIP). 

Work is so much fun! I have always loved throwing myself into my work, figuring things out as I go, connecting with my amazing co-workers, chatting about geeky TV shows + movies my wife and I are watching. Best of all? Everybody in the company works from home. I feel like I am so much more productive without having to slog through an hour long commute to an office where I am on Zoom all day. 

It was a panicky three month job search when my development team at MassMutual -- where I had been the past four years -- all received notice back in April 2025 that our team would be phased out in July. It was a rough job search, with next to no interviews lined up.  Automation with Capybara and Watir, Ruby-based test automation solutions I've been using for six years just aren't as popular anymore. If it wasn't for a hiring manager personally recruiting me for his company, who knew me back when I was running the Boston area Meetup for the UK-based Ministry of Testing, a software testing community, I may still be unemployed.

The best perk of all? I was hired just in time to join SELF's yearly off-site in Puerto Rico! Sipping rum & cokes by the beach at the Caribe Hilton with my new co-workers face-to-face, on only my second week at the new company, was amazing!

I am completely astounded at my good fortune, and I hope I am not going to be job searching for a long, long time. 

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

New Playwright Course: Playwright Essential Training by Butch Mayhew

Butch Mayhew, Playwright Ambassador and author of the LinkedIn Learning course, Learning Playwright, has just released a companion course, Playwright Essential Training: Abstractions, Fixtures and Complex Scenarios. 

Where Learning Playwright covered how to install Playwright, create simple functional, AI, and visual tests, how to maintain tests and how to scale them, Playwright Essential Training dives a lot deeper. It goes over topics such as:

• How do you set up page objects in Playwright?
• What does test data management look like?
• How do you mock out test data?
• How do you deal with tricky components such as alerts, dialog boxes, and multiple windows?

Thank you very much, Butch! Looking forward to taking the course!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Time to Practice Playwright!

Back job searching again? Time to learn a new automation toolset!

Thank you, Butch Mayhew for your course, Learning Playwright on LinkedIn Learning.

Playwright was launched back in January 2020, by Google Puppeteer developers who took what they learned to build a new automation framework for Microsoft. According to the Playwright Doc, "Getting Started", "Playwright Test was created specifically to accommodate the needs of end-to-end testing. Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox. Test on Windows, Linux, and macOS, locally or on CI, headless or headed with native mobile emulation of Google Chrome for Android and Mobile Safari".

Related Links:

• GitHub: https://github.com/microsoft/playwright
• NPM: https://www.npmjs.com/package/playwright
• Documentation: https://playwright.dev/docs/intro 

So far, I am halfway though Learning Playwright. I am really loving how the course has it's own GitHub repo, with each chapter has its own branch. Students can practice building out a Playwright framework, chapter by chapter, typing out the code themselves -- the best way to learn. And if the student feels like skipping that exercise they can just pull the next branch lesson to their local machine.

Butch included a Resources File which walks through things like:

• Learning Playwright
• Node Prebuilt Installer
• Installing VS Code
• Playwright Test for VS Code

The Resources File also covers: Updating The Playwright Config, Configuring Browsers, Exploring The Test Runner Command Line Interface, Exploring the VS Code Extension, Exploring Playwright UI Mode, Generating Tests With Codegen, Overview of Assertions in Playwright, Handling Cookie Authentication in Playwright, Visual Testing in Playwright, API Testing in Playwright. It also covers using  Playwright Screenshots, Videos, and Reporters, the Playwright Trace Viewer, and Scaling Playwright Tests.

Butch, a Playwright Ambassador, also has a related site, Playwright Solutions. 

Four years ago, it was Ruby that was in demand, an easier language for manual testers to pick up. I was able to go from an SDET position using Ruby + Capybara to an SDET position using Ruby + Watir. Now, it seems everyone is looking for Playwright + TypeScript... which I do not have any experience in.

Looks like I need to take a few more courses, then practice putting together some Playwright frameworks, blogging here about my Adventures in Automation.

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Test Guild: Building a Career in QA with Judy Mosley

I absolutely loved this week's episode of Joe Colantonio's Test Guild, Building a Career in QA with Judy Mosley.  

This week, Joe interviews Judy Mosley, a QA Engineer of seven years who shares the fun and excitement of testing, discovering the field after attending a web development bootcamp. Anyone who is a tester should listen to it.

I also love her blog, "Failure is Feedback", where she elaborates on her story with the entry "Quality Insight: Why 🫵 Should Share Your Career Journey".

... It's so hard keeping up with a blog. It was easier when I was at ThreatStack, writing official articles on the company website about how I was testing software there. While at MassMutual, I was putting my energy writing internal Confluence documentation on how to to install and run the framework I was building, and less on this blog.  

Thank you so much, Judy for sharing your story!

So, are you a software tester? Have a blog of yours to share? Leave a link in the comments below!


Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

When building out automation frameworks, keep the stakeholders in mind

When building out an automation framework, I always try to keep the wants and needs of the stakeholders in mind.

Business analysts want to know if we have tested the product against the business requirements.

QA Managers want easy-to-read reports that spell out what is passing and what is failing at a glance.

Developers may want to check for sporadic errors that randomly pop up.

Testers want to have regression tests automated so they can focus on testing the new features... and may want a library of building blocks so they can put together their own tests.

A prioritized backlog of automation JIRA tasks keeps everything on track, and end-of-sprint demos gives the dev team a chance to chart the progress and tweak what they see.

Being embedded in a development team, I can build and customize an automation framework, one two-week Agile sprint at a time.

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

How automation recaptures the magic of testing

There is something magical that first time a software tester explores a web application.

A head full of questions, they fire up their favorite browser and try to figure out how how the app works. How the app behaves when you enter unacceptable parameters. How the app performs under stress. And they will happily take notes about their discoveries as they test their own assumptions how it should work until their curiosity is satiated.

Every subsequent time they view that same app, some of that magic is lost. The unfamiliar becomes the familiar becomes the routine which becomes boredom.

That is where I come in. "My job is to make your job easier", I always say to our test lead, John Jurek.

What is imperative that needs to work before we begin testing? Make that a smoke test.

What test script is so mind-numbingly boring to manually execute? Let's add that to the backlog of regression tests to write.

Sure, it might be fun to run a test once or twice, but once you run a test seeing how one part of a feature behaves in Chrome, Firefox, Safari, Microsoft Edge, and all the little mobile widths a responsive web application can have, the fun slowly disappears.

Keep the testers focused on testing new features. Farm out to the automated tests the rest. That helps keep the magic of testing alive.

... And once you have an automated framework stood up that fits the needs of the development team, you can start training the testers on writing their own automation to investigate the web app.

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Need a SDET with 10 years experience for a Remote / Hybrid position in Boston?

My development team and I at MassMutual were all just given notice that our last day would be July 11, 2025. Anybody looking for a Software Developer in Test with ten years of experience? 

I've been trying to sum up what I have been doing for almost four years at MassMutual. How does this sound?

MassMutual - Boston                                    Sept 2021 - July 11, 2025

• Shaped automated test suite to the wants and needs of the stakeholders on the Payment Experience team. Drafted UI + API positive and negative tests written against reviewed business requirements of new Billing & Payments features being developed that sprint. New and existing smoke and regression tests were executed in the developers feature branch, testing their code before it was ever deployed to the DEV or QA environment. 
• Designed browser tests in Cucumber GIVEN / WHEN / THEN format so user data was reset to the correct state using MassMutual APIs before the browser ever opened: Pending payments cleared, Premium and Loan AutoPays turned off, and Billing & Payment API was 200 OK. 
• Created UI and API tests in Ruby, Watir, Applitools, to be run on the DEV + QA environments through Jenkins jobs running daily smoke test and weekly regression tests. BrowserStack + LambdaTest, Chrome, Firefox, Safari, MS Edge, Chrome Mobile, Android and iOS. 
• Triaged daily smoke test run to determine the health of the environment before testing.
• Mentored manual testers in writing automation code, tailoring the README and Confluence documentation to their learning needs, whether setting up their local environment running the smoke + regression suite, or how to use the automation steps drafted in Cucumber/ Gherkin.

Happy Testing!

-T.J. Maher
Software Engineer in Test

Twitter | YouTube | LinkedIn | Articles

Stepping down from being a Meetup Organizer of Ministry of Testing - Boston

Today is a hard day. I just stepped down from being a Meetup Organizer of the Ministry of Testing - Boston. Life has been so busy the past year, I haven't even been able to host even a virtual Lunchtime Discussion.

I first encountered Conrad Holloman's Greater Boston QA and Testing Meetup when he hosted a Meetup event back in 2016 when Fitbit - Boston, where I was a budding automation developer. Before I knew it, I was organizing events, volunteering to be on discussion panels, and giving presentations to the group. When Conrad moved the Meetup under the Ministry of Testing banner at the end of that year, he encouraged me to take over as organizer.

As Meetup Organizer, I have had so much fun recruiting speakers such as Angie Jones, Seb Rose, Matt Wynne, Lisa Crispin, Moshe Milman, Eran Kinsbruner. Thank you so much, SmartBear and Mabl for always hosting us!

Everything came to a halt due to Covid. I tried switching to virtual Meetup events in 2022 for this group, but that came to a halt when my kiddo was diagnosed with ADHD + Autism. All of my time and energy from that time forward has went to my family, shuttling my son to new schools and doctor appointments, and it hasn't let up.

I have really enjoyed being a Meetup Organizer! It has been a thrilling ride!

-T.J. Maher

MassMutual Boston, Software Engineer in Test
Meetup Organizer, Ministry of Testing - Boston

Twitter | YouTube | LinkedIn | Articles

The Cheezy Internet: Composing tests using Cucumber and Gherkin

This is fourth in a series of blog posts. Care to go back to the beginning?

Following along with Jeff "Cheezy" Morgan's eBook, "Cucumber and Cheese: A Tester's Workshop" (2017), after setting up a development environment, we started creating in Ruby + Watir the basic building blocks for an automated test framework, as we saw in the last Adventures in Automation blog entry. 

Instead of going into great detail testing against the complex test site Jeff uses in Chapter 4: Cucumbers and Puppies of is book, with Sally's Puppy Adoption Agency, we are using the simpler test site, The-Internet https://the-internet.herokuapp.com/login, by Dave Haeffner. 

This blog post will explore scaffolding a site and composing acceptance tests using Cucumber and Gherkin. 

Scaffolding a Site Using TestGen

It's always a challenge for me when creating a new project to figure out where everything should go. What should the folder and file structure be? Luckily, Jeff Morgan created a Ruby gem called "testgen" which solves all of these problems. 

• Change the directory to the src folder in your home directory: cd ~/src
• Pick a name for your project, such as "cheezy_internet"
• Install Jeff Morgan's Ruby gem testgen on your local machine: gem install testgen 
• Use the Ruby gem testgen to create a new file hierarchy: testgen project cheezy_internet 

The Cheezy Internet: Writing Ruby + Watir tests for The-Internet

This is third in a series of blog posts. Care to go back to the beginning?

Last blog entry, based on Jeff "Cheezy" Morgan's LeanPub book, "Cucumbers and Cheese: A Tester's Workshop", we set up our local machine, wrote and executed our first Watir program. 

Based on his book, we are going to attempt to sketch out and write our first test against Dave Haeffner's sample login page on The-Internet at https://the-internet.herokuapp.com/login.

The-Internet / Login Page

Drafting a Test 

A sample test for The-Internet / Login could be:

• Go to https://the-internet.herokuapp.com/login
• Enter tomsmith into the username textbox.
• Enter SuperSecretPassword! into the password textbox. 
• Press the Login button. 
• If the Secure Area page does not show "'Welcome to the Secure Area", fail the test.

The Cheezy Internet: Installing Ruby + Watir + Chromedriver + VS Code on Windows 10

This is second in a series of blog posts. Care to go back to the beginning?

For this exercise, we will be setting up a Windows 10 box for use as a development machine.

Macbooks are amazing machines. I love coding on them. Using the Mac Terminal in Homebrew. Dressing them up with decals, such as the glasses-and-tie decal my wife bought for my very first work laptop back at Fitbit-Boston.

Love dressing up the Apple logo!

I definitely would not recommend using your work machine, if you have one, for anything but work and work-related activities. 

The problem is that when it comes to non-work related programming activities, it can leave for me, well, less glamourous and sparkly alternatives.  

My home computer always has been a Windows Desktop, a big ole workhorse to store the family photos and videos, ancient video games like the original StarCraft, old blog posts, articles, and essays. But they can be so hard to set up sometimes with the toolsets I need to put together an automation framework. 

Based on the information in Jeff "Cheezy" Morgan's LeanPub book, "Cucumbers and Cheese: A Tester's Workshop" (2017), we will start setting up our local machine. 

The Cheezy Internet: Creating a Ruby + Watir + Cucumber automation framework with Jeff Morgan's Page-Object gem

Greenfield projects have been rare in my automation development career. The programming language, the toolsets that help craft the automation frameworks, the methods that interact with the pages of the web  application under test, all have been selected long before I joined the software development team. 

My job as an automation developer is to get up to speed on the automation framework as fast as possible. To climb the steep learning curves quickly, I have been creating little toy projects, ever since my first automation development job at Fitbit-Boston back in 2015. Doing this, I get explore the tools and technologies used for automated testing, using similar source material testing against a simple test site.

Our source material for this current exercise will be Jeff "Cheezy" Morgan's LeanPub book, "Cucumbers and Cheese: A Tester's Workshop". Even though "Cucumbers and Cheese" was written in 2012 and last updated in 2017, it contains a treasure trove of information for beginning automation developers, walking the reader through building an automation framework in Ruby + Cucumber + Watir + Jeff's page-object Ruby gem. Jeff's application under test is a mock site, the Puppy Adoption Agency at http://puppies.herokuapp.com/

Our application under test for this project will be an old favorite of mine: The-Internet at https://the-internet.herokuapp.com/, created by Dave Haeffner. Dave Haeffner is the author of the SeleniumGuidebook.com -- which comes in C#, Java, JavaScript, Python, and Ruby editions -- and the very helpful blog ElementalSelenium.com.

Next week, we will start using the information in "Cucumbers and Cheese" to start building an automation framework, starting with setting up a test environment. 

NEXT -> Setting up a development environment on Windows 10

Happy Testing!

-T.J. Maher
Sr. QA Engineer, Software Engineer in Test
Meetup Organizer, Ministry of Testing - Boston

Twitter | YouTube | LinkedIn | Articles

Just purchased a new automation development book, Cucumbers and Cheese: A Tester's Workshop

This morning, I just purchased Jeff "Cheezy" Morgan's LeanPub book, "Cucumbers and Cheese: A Tester's Workshop" https://leanpub.com/cucumber_and_cheese . 

Although this book came out in 2012 and was last updated in 2017, the Gherkin + Ruby + Watir + Page-Object Ruby Gem framework described was the basis of the initial framework my workplace uses. 

Related Links: 

• Jeff Morgan's GitHub: https://github.com/cheezy
• cheezy/page-object: https://github.com/cheezy/page-object

... Can't wait to read it! 

About the Book:

"In Cucumber and Cheese, Cheezy shares the concepts that help teams succeed with ATDD (aka

BDD, story test-driven development, specification by example, business-facing tests that guide

development, take your pick). And he does this the best possible way - with examples. You can

work the examples along with Cheezy, who makes them accessible even to people with not-so-much

coding experience.

"Cucumber and Cheese isn’t just for web developers. The book works through examples for web

services, mobile apps, XML, native windows apps. You’ll even learn how to create test data by

example.

"The book includes essential seekrits such as how to handle Ajax and that pesky waiting for events

to occur. Cucumber and Cheese is soup to nuts, from general principles like page objects and Don’t

Repeat Yourself, to specifics for Rails developers on RSpec matchers and ActiveRecord reference

material.

"Cheezy has a gift for making you enjoy yourself so much, you might forget you’re learning

something you thought would be hard. Lots of examples! And puppies!"

- Foreward, by Lisa Crispin, Coauthor of Agile Testing: A Practical Guide for Testers and Agile Teams

About the Author: 

https://www.industriallogic.com/people/cheezy/

"Jeff “Cheezy” Morgan has been teaching classes and coaching teams on agile and lean techniques since early 2004. Most of his work has focused on the engineering practices used by developers and testers. For the past few years, he has experienced great success and recognition for his work in the area of Continuous Delivery.

"He is based out of Toronto and focused on bringing Industrial Logic’s passion for technical excellence, modern agile, and lean product management to the Canadian market. He is the author of Cucumber and Cheese, A Testers Workshop.

"You can meet him at most of the conferences around North America during the year or follow him on Twitter @chzy".

Happy Testing!

-T.J. Maher
Sr. QA Engineer, Software Engineer in Test
Meetup Organizer, Ministry of Testing - Boston

Twitter | YouTube | LinkedIn | Articles

Long time no hear!

Hey there! It's been a while. How have you been? 

It's been a rough year, hasn't it? 

Since we last talked, a global pandemic started, I lost a job, found a new job, found a new house, moved into the new house, got vaccinated for COVID-19, helped my in-laws move in, found out the new job wasn't really an automation development job, quit that job, the family caught mild cases of COVID-19, and started a brand new job last week.

There just hasn't been any mental energy left over to blog. 

When my three year old son has been in daycare, I am in work-mode. 

When my son is home, I have been in Dad-mode. 

I just haven't been in blogger mode ...

... Until now. 

I think life is more settled now. 

I think I can start blogging again, now that I have an automation development position to blog about.

See you online! Feel free to ping me on Twitter at @tjmaher1.

And, as always .... Happy Testing!

-T.J. Maher
Sr. QA Engineer, Software Engineer in Test
Meetup Organizer, Ministry of Testing - Boston

Twitter | YouTube | LinkedIn | Articles