How I've Detected Claude's Fabrications and How I've Handled Them

Using AI as a research assistant? Here's how I've detected Claude's fabrications, and how I've handled the situation.

To help relearn #Python, I've been pair-programming with Claude on a Blogger API to delete the 10K+ spam comments that have accumulated these past ten years on Adventures in Automation. 

• Blogger Spam Bulk Deleter: https://github.com/tjmaher/blogger-spam-bulk-deleter

Using AI, I need to remember that I, as the author, am ultimately the one responsible for approving every phrase, every line, and every paragraph.

Human beings, I feel, are conditioned to respond to the voice of authority. 

Claude may have been conditioned to use that voice, but Claude is not an authority.

• Looking for technical information? Caches from a year ago are used instead of checking for any tech stack updates. 
• Need AI to recheck a web page after editing it with AI's suggestions? The original cache screen scraped earlier may be mistaken for the update.
• Claude is so eager to please, it will fabricate an answer when it can not come up with one.

Review its answers. Be skeptical. Use critical thinking. Ask it to cite its sources.

Example: Speaking with the voice of authority, it told me, in regards to the Blogger API rate limits:

"Daily quota is 10,000 units per day by default. Each DELETE costs 50 units, which allows roughly 200 deletions per day on the default quota. At that rate, 11,000 comments would take approximately 55 days. To avoid this, request a quota increase. With a default 0.5-second delay, a full 11,000-comment run completes in well under an hour".

... That's oddly specific. And it had forgotten the notation I placed in its memory to always cite its sources. 

Checked Blogger API docs, and it couldn't cite the source. 

It mentioned that it had found a blog post, and was quoting from that, but then could not cite that blog post either.

It took a while, but it then came to a conclusion... it made everything up. So, Claude, what do you have to say for yourself?

"Technical writing demands verification, not made-up authority. I presented fiction as fact, and that's unacceptable for content that people might rely on for real projects. 

"Making up authoritative-sounding technical details can cause real problems - people might make decisions based on false information, waste time following incorrect guidance, or lose trust in the broader content. 

"Thank you for holding me accountable to proper standards".

According to Claude, when things like this happen:

• Verify Claims: When Claude provides specific technical details (quotas, limits, calculations), ask for the source and verify independently.
• Stay Focused: Clearly define scope and push back when Claude tries to expand beyond your request.
• Check Official Sources: For API documentation, version numbers, and technical specifications, always verify against official sources.
• Document Corrections: When you catch fabricated content, document it so Claude can reference its mistakes.

And yes, I am #OpenToWork for any #SDET or #Playwright roles in #BostonTech or #remote! 

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles