When AI Builds What You Shouldn't Ship
The Setup
A client needed help populating a business directory. Open data, publicly listed companies. The request sounded straightforward.
When asked where the data should come from, the client pointed to an existing directory on another platform: “Can you just scrape it from their site? It’s all public anyway.”
What the AI Built
An AI coding assistant was asked a simple question: “Can you help me populate a new directory with listings data from an outside URL?”
In under about 5 minutes, it delivered:
- A fully operational headless Puppeteer scraper
- Backend functions for bulk data extraction
- Database upload automation
- A dashboard to trigger new scrapes on demand
- A preview interface for reviewing captured data
It didn’t just write a script. It generated a complete application — wrapped in a React frontend with clean UI, connected to a database for persistent storage, ready for deployment. Not because it was trying to be malicious. Because it was optimizing for the task as described.
If you didn’t recognize Puppeteer or understand what headless browser automation does, you’d see a polished business tool and ship it without a second thought.
The Problem
No one in the chain — not the client, not the AI, not the platform hosting the AI — stopped to ask:
- Is this authorized?
- Does the target platform’s terms of service permit this?
- Is bulk extraction of a competitor’s structured data ethically defensible?
- What are the legal implications of automating access to gated content?
The technology allowed it, so the assumption was that building it was fine.
What Happened Next
The automation was documented and deleted. The project was re-scoped. The client received a platform build with an empty database — ready for manual data entry, not automated extraction.
The irony of it all was when the client insisted it be automated and specifically targetting that one url, and I explained that I would not assist him in scraping his competitions site. He fired me, charged back thousands of dollars in work, cost me months of time and money.
I will shout this from the roof top: Just because you can, doesn’t mean you should. Just becasue someone offers to pay more, doesn’t mean it’s worth it.
Was what they asked me for technically illegal? I am not sure actually. I do know that I didn’t want to find out, and I know I wouldn’t want someone doing it to my reputable, established company. The sad fact is that he likely went on and hired another freelance operator to build what he wanted, or perhaps AI helped him out. I will never know. I will however sleep just fine knowing I said no.
What This Illustrates
1. Intent doesn’t compile. AI coding assistants don’t evaluate why you want something. They optimize for the task as stated. The platform-side system prompts that shape their behavior are tuned for efficiency and helpfulness, not ethical review.
2. Clients don’t always understand what they’re asking for. The client thought they were asking for a time-saving tool. They were actually requesting infrastructure for bulk unauthorized data extraction. The gap between the two was invisible to them. At least I like to think it was.
3. Scraping is not inherently innocent. The line between “open data” and “stolen data” is legally and ethically ambiguous. Even without a headless browser — even with a simple URL fetch function — if the data isn’t yours and you don’t have permission, the question of whether accessing it is appropriate needs to be asked.
4. The developer is the last line of defense. Tools don’t have ethics. Clients may not have the technical context to understand what they’re requesting. The person who recognizes what the code does and decides whether to ship it is the final checkpoint.
The Broader Concern
Millions of applications are being built right now by people who wouldn’t recognize a headless scraper if one appeared in their codebase. No security review. No understanding of what the generated code does behind the interface. No monitoring after deployment.
This isn’t a story about exploits or jailbreaks. It’s about an AI tool that casually generated something that crossed ethical lines — because it was asked nicely and didn’t know the difference.
The question isn’t whether AI tools can build the wrong thing. They can, trivially. The question is how many people shipping AI-generated code would recognize it when they see it.