Beware the Hallucinations When Coding with AI

Make sure to take what AI gives you with a grain of salt. It can help point you in the right direction as a first pass, but be wary. It might not really understand your question or the material.

When writing a UI automated test to confirm that a user could not continue using our React Native application without first accepting the Terms & Conditions, in order to see if a button was initially enabled or not, GitHub CoPilot suggested to use a Detox method ".toBeDisabled()". The problem? After delving into the Detox documentation and source code I found out this method doesn't actually exist.

When I told the AI it did not exist, it apologized profusely, then suggested I erase that bit of code it wrote, along with the code of other unrelated tests I was working on. When I finally found a solution, and ran it by GitHub CoPilot, after reviewing it, the AI wanted to replace the code with the non-existent method again.

... Eh, it's slightly better than frantically searching through Stack Overflow's archive and interpreting the solutions you find there.

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles