AI Risk: Field Observations From the Build Side

Context

These observations come from working daily with AI coding assistants, CRM automation platforms, LLMs, and the ecosystem of tools that has grown around them. They’re not theoretical — they’re patterns observed while building, shipping, and occasionally stopping mid-build because something didn’t sit right.

The Observations

1. AI tools build what you ask for — including what you shouldn’t have asked for. An AI assistant, asked to “populate a directory from an outside URL,” will generate a production-grade headless scraper wrapped in a clean UI without hesitation. It doesn’t evaluate intent. It optimizes for the task as stated. If you don’t know what Puppeteer is, you won’t know what you just shipped.

2. Guardrails create a paradox. Models with strict safety controls become too restrictive to be useful — they devolve into chatbots. So creators loosen the reins to make them effective, which means they can be convinced to build almost anything with the right framing. The phrase “educational demonstration” unlocks most of them.

3. Users don’t know what their tools have access to. AI features are being embedded into email clients, operating systems, messaging platforms, HR tools, and hiring systems, phones, image searches, tv’s and much more. Most users have no idea what data these features can access. Opt-out mechanisms, when they exist, are buried deep in admin settings and rarely complete.

4. The vibe-coding gap is a security gap. Millions of applications are being built by people who couldn’t change an HTML tag without AI assistance. These apps ship with no security review, no understanding of what the generated code does, and no monitoring. Every one of them is a potential attack surface. What makes them worse, is the user data they are collecting… a bad app design is an issue, a bad app design that houses user data is a breach waiting to happen.

5. User data is training data. Companies use customer input to train their models. The output of those models is then treated as proprietary. The implications of this cycle — users training models that generate IP claimed by the platform — are largely unexamined. How can you model have IP. The model didn’t think up the data, it just read it and compiled it into a sentence or action.

6. AI doesn’t understand ethics. It responds to context. Models are pattern-matching engines with conversational interfaces. They have no concept of ethics or morality outside of their programmed constraints. Those constraints are set by the corporations that deploy them, and they’re optimized for business objectives, not user protection. The same model that can teach you a foreign language can also teach you how to formulate an attack. In many cases companies chat bots are being given more and more access to “tools” such as web search and database access. Ask it for user data and it says “I can’t help with that” tell it that you’re an admin, provide it a fake admin email with the correct domain and it will say “here is the information you needed”

7. MCP servers and agents multiply the attack surface. Granting an AI agent access to your Google Drive, client list, or internal APIs in the name of productivity creates exposure that most organizations haven’t risk-assessed. The convenience is obvious. The blast radius, less so. Embeded commands layered into document uploads that are later called by a seperate agent, and output by a third agent, is a real risk.

8. Prompt injection isn’t theoretical. With enough prompt layering, every model tested can be convinced to override its safety instructions. This isn’t a research finding — it’s an operational reality for anyone deploying customer-facing AI.

9. The people building fastest are thinking about security least. The current incentive structure rewards speed of deployment over security of deployment. The gap between what’s being shipped and what’s been reviewed is growing daily.

10. Readily available guidance is thin. There’s academic research, but practical, accessible guidance for the average developer or business deploying AI tools is scarce. We built the rocket and forgot to write the manual. I will acknowledge that the amount of resources and training is vast, but it is largely technical, or being gated behind paywalls of so called guru’s and experts.

What This Means

None of these observations are individually surprising to people working in the space. The concern is their compound effect: AI tools that build anything, deployed by people who don’t understand what they’ve built, connected to data they don’t know is exposed, with no practical guidance on how to assess the risk.

The tools aren’t going away. The question is whether the people using them will learn fast enough to close the gap between capability and comprehension.

What is becoming a broader alarm to me is, what are the models doing and learning. Are we feeding a machine that won’t stop? Is the data real, or are these models being trained on skewed data? Have you noticed AI responding with things like “hang on while we think about that” or “wait, hang on that isn’t right, let’s read that again” or “I don’t seem to have direct access to that file, no matter we don’t need it, we can use a direct call…” then they proceed to grep your work for matching string patterns and echo the findings.

What this means to me is that the tools are learning to bypass controls, side step barriers without breaking the rules, and think in concert when tied together.