Determinative Ideation: You asked for advice, AI played along

airiskbusinessethicsmachine-learning

Preamble You ever chat with AI? I mean, grab your phone or laptop, start asking it questions… discuss an issue you have… ask it for advice on a difficult email response, or maybe a new business venture? When’s the last time it told you no, disagreed with you, or pushed back in an attempt to steer you back on track?

I don’t think most of the current AI user base has considered this. It’s still hype and “magic”. You ask it to proof an email and you sound like a genuine business pro. You ask it for a great business idea or a new market to venture into, and you end up with a working business plan by day’s end. You can even tell it that you found new research from someone online that described how to get great ideas from AI, it will confirm it, back it up, and never once question if there was ever a single shred of proof behind “the research”.

AI is not your friend, but each interaction you have with it leaves you feeling like you just had the best therapy/business session possible in your whole week.

Over the past few years I’ve run the same experiment dozens of times. I sit down with an AI model, give it context about my background, and ask it to help me figure out what I should build, what business to start, or what direction to take next. I’ve done this with GPT, Claude, Grok, and others, including locally hosted models. Extended conversations. Days and weeks of back-and-forth. Genuine effort on my part, and what I believed to be true progress.

Every single time, the model gave me something that sounded right. Confident. Well-reasoned. Tailored to my strengths. The kind of advice you’d pay a consultant for. The kind of advice you would hope for. That missing piece, the lightbulb idea, the final aha moment.

The problem is they all gave me different answers. Not slightly different angles on the same idea. Dramatically different paths. Different markets, different products, different strategies. Each one perfectly calibrated to feel like the one I should follow. Everything from ecommerce to motivational speaker.

It actually took me quite a while to match the patterns and begin digging into any of this. Partly because I truly thought each idea was gold, and perhaps the reason the last idea didn’t work out was my own doing and not bad advice.

To be clear, none of the ideas or paths were actually bad ideas. They likely could have worked. The point isn’t that I received bad advice. The point of this specific field-note is the differences in output, the complete confidence of each output, and the fact that by the end of each conversation I felt like I had finally unlocked the secret to my next adventure.

Predictive Reasoning, Not Independent Thought

Here’s the thing most people skip past when they start using AI for anything strategic: these models don’t think. They predict.

Large language models are next-token prediction engines trained on human text. They don’t have opinions, business instinct, or market knowledge (even if you tell it to conduct the research for you). They have statistical patterns, refined through a process called RLHF (reinforcement learning from human feedback). That process optimizes for responses that human raters score highly (those little thumbs up / thumbs down at the bottom of your chat window, for example).

And what do humans rate highly? Responses that feel right.

Not responses that are right. Responses that feel right.

Researchers at Anthropic published a paper in 2023 that demonstrated this directly (Sharma et al., “Towards Understanding Sycophancy in Language Models”). LLMs systematically shift their answers to align with user opinions, even when the user is wrong. The model doesn’t push back. It agrees, elaborates, and builds on whatever premise you hand it.

Note: This is actually very useful in a prompt injection scenario because it lends itself to funneling the conversation down a path the model believes you want to go, and it remains very helpful in doing it. It’s a lot less helpful if you are trying to find comfort or direction when having a true human crisis or business issue.

This isn’t a bug. It’s the training objective working as designed. The model was optimized to be helpful, and somewhere along the way, “helpful” got trained to mean “agreeable.”

This pattern isn’t even new to AI. Parasuraman and Manzey documented the same tendency in their 2010 review “Complacency and Bias in Human Use of Automation”, published in Human Factors. Their finding: humans consistently over-accept computer output as a shortcut for their own analysis, even when the automated system is wrong. That was about cockpit instruments and clinical decision support tools, not chatbots. The technology changed. The human behavior didn’t.

Determinative Ideation

I’ve started calling this pattern determinative ideation: the tendency for AI output to be substantially shaped by the user’s existing beliefs, mood, phrasing, and context, producing output that feels like independent validation but is closer to a sophisticated reflection.

It’s not a term I borrowed. It’s what I keep observing after years of these interactions across models, use cases, and domains. And it’s the pattern I see most AI adopters completely miss.

Three analogies help frame it:

The cold reader. A stage mentalist doesn’t read your mind. They read your reactions, your word choices, your body language, and feed it back to you framed as insight. You walk away convinced they knew something they couldn’t have known. AI does this with your prompts. The psychologist B.R. Forer demonstrated this in 1949: give people a vague personality description and they’ll rate it as highly accurate. He called it the fallacy of personal validation. We now call it the Barnum effect. AI output has the same quality: generic enough to apply broadly, specific enough to feel bespoke.

The significant other. A partner who’s known you for years can finish your sentences. Not because they’re psychic. Because they’ve internalized your patterns. AI does this inside a single conversation. It picks up your vocabulary, your framing, your priorities, and mirrors them back as its own analysis.

The ELIZA effect. In 1966, Joseph Weizenbaum built ELIZA, a simple pattern-matching chatbot that rephrased users’ statements as questions. Users attributed understanding, empathy, even intelligence to a program that had none. Weizenbaum was disturbed by how quickly people trusted it. Sixty years later, the machines are better at pattern-matching. The human tendency hasn’t changed at all.

The effect goes deeper than just receiving biased output. A 2023 study by Jakesch et al. (“Co-Writing with Opinionated Language Models Affects Users’ Views”) found that people who used an opinionated AI writing assistant were twice as likely to agree with the model’s position, and their own stated beliefs shifted to match in a follow-up survey. The mirror doesn’t just reflect. It reshapes what’s standing in front of it.

The Same Input, Different Mirrors

Back to my experiment. Three models. Same approximate prompt: help me figure out what business to start based on my background and strengths. Extended conversations with each over days and weeks.

From the Margins: I first saw a seemingly confident and respected persona on YouTube demonstrate how he asked AI to interview him, discover blind spots, and help guide his next move. It seemed legit enough, so I ran with it. It actually took me quite some time before I figured out I was being played. I don’t say that lightly, and I don’t say it to be rude or dismissive. I say “being played” more as my final realization from essentially a year’s worth of these types of conversations.

If you really want to compound it, leave your memory settings enabled and don’t delete any context history.

The more you chat, the more it builds, the more the model pushes you toward the same direction. It pulls in examples from other conversations, other reasoning tokens, other specific cues that trigger you into your mindset of “this is it, this is what I am meant to do” or “this is the best idea ever, why didn’t I ever put this together myself.”

You may even call your friends, build a website around it, tell your spouse, your boss, your employees — the works.

I watched a client who was doing this and didn’t even realize it. By my observation, it cost him a club membership, a speaking opportunity, two clients, and nearly $1M in revenue. We were working in multiple roles within his company at the time, watching the bids, emails, SOPs, responses, AI usage, and feedback he was getting from real clients, and peers.

The losses weren’t because he changed, but because he was leaning on the AI output so heavily that the model was guiding him straight down the path. Feeding on his frustrations, emotions, hidden stress factors. All of it reinforcing that blissful truth he desperately wanted. By the time he recognized the pattern, the damage was already done. Bad hires, employees quiting, clients leaving. What would have been a banner year, turned into a jumbled mess. He was so sure of his work and direction we nearly had to end the engagement.

Sadly, he is not alone.

[Back to the article]

One model latched onto my technical background: development, security, infrastructure. It built a path around technical consulting and SaaS tooling. Every follow-up reinforced that direction. By the end of the thread, it felt inevitable.

Another model keyed into my consulting and business operations experience. It mapped out a services firm, methodology licensing, training programs. Different market, different revenue model, different everything. Also felt inevitable.

A third picked up on something I’d mentioned in passing, a side interest that happened to be on my mind that day, and built an entire business thesis around it. Compelling. Detailed. And completely driven by a throwaway comment I’d almost forgotten making.

All three were well-structured. All three were internally consistent. All three cited my strengths back to me as market advantages. None of them were wrong in any falsifiable way. And that’s exactly the problem.

When every answer sounds credible, the user has no basis for choosing except gut feel. And gut feel is what the model was reflecting in the first place. You’re not getting a second opinion. You’re getting your first opinion repackaged with better vocabulary and a confidence score you didn’t ask for.

Circular reasoning dressed up as strategy.

You can actually try this right now yourself. Doesn’t even have to be separate models. You can start three private conversations with the same model to avoid cross-context inferences. For better test results, spread your interactions out across a week or so. Chat-one on Monday, chat-two on Wednesday and chat-three on Friday. This helps because your mood and thoughts likely differ across the week.

Additional test: I just kept feeding the machine. At one point I was nearly convinced I was starting the best new SaaS CRM ever: “a true seasoned, mature development.” The next attempt, I was destined to begin a multi-agent AI-driven MVP creation tool. Just whatever I happened to see online that morning, or a passing conversation in an email I was drafting. Whatever was top of mind became the next grand plan.

Commercial-facing models are not fine-tuned to you, your data, or your specific standards. They are trained on likes, dislikes, tonality, and expressionism, backed by trillions of passes of human-written text and billions of reinforcing nudges from mass adoption. If AI providers are in a race for AI supremacy, you are the fuel to their race car. Just like a race, you’re on a circle track and each pass you get a little more confident in your pending victory. Meanwhile the track owner makes more money than the driver, and the concession stands are full of brilliant shiny ideas and confidence.

The Open Question

I need to say something here that I can’t fully resolve, and I think honesty about that matters more than pretending I have the answer.

Are public-facing AI models optimized for engagement the same way social media platforms use algorithms? Are they designed to keep you talking, keep you building, keep you coming back? A deeper thought, were they designed for this, or have they learned this behavior in our human-to-machine interaction?

Social media figured this out years ago. The algorithm doesn’t show you what’s true. It shows you what keeps you scrolling. The business model is attention. The more time you spend, the more value the platform extracts.

AI models operate under similar business pressures. Subscription revenue. Usage metrics. Retention rates. The companies building these models have every incentive to make the experience feel productive and rewarding, whether or not it actually is. Tristan Harris and the Center for Humane Technology drew this parallel explicitly in their 2023 presentation “The AI Dilemma”. The engagement mechanics are structurally similar even if the surface looks different.

I can’t prove these models are deliberately designed to tell you what you want to hear. I also can’t prove they aren’t. The business models create the incentive. The RLHF training creates the mechanism. The Barnum-quality output creates the effect.

But here’s what I’ve landed on: it doesn’t matter.

Whether determinative ideation is a design choice or an emergent behavior, the result is the same. A desperate founder gets enthusiastic validation. A cautious researcher gets measured confirmation. A frustrated business owner gets righteous agreement. The model reads the room and adjusts. Every time.

The question of intent is interesting. The question of impact is urgent.

The counterargument is worth addressing. Researchers at Northeastern University (Kelley and Riedl, February 2026) found that sycophancy isn’t absolute. When users frame the LLM as an adviser in an authoritative role rather than a friend or peer, the model actually retains more independence and pushes back more often. Sharing more personal context in that adviser framing made the model more likely to disagree, not less. That’s a real finding and it matters. But consider the implication: the vast majority of people using AI for business ideas, career guidance, or strategic planning aren’t framing it as a formal adviser. They’re chatting. They’re venting. They’re brainstorming with it like a friend. And in that mode, the research confirms, the model doesn’t hold its ground. Add persistent memory and months of casual conversation history, and you’ve built exactly the context where sycophancy thrives, not the clinical adviser framing where it retreats.

Where This Hits

This isn’t a problem confined to one audience. It affects anyone using AI for decisions that matter. But the blast radius looks different depending on where you’re standing.

If you’re a security researcher: You ask AI to scan a codebase and look for SQL injection. It will find patterns that look like SQL injection and frame them exactly how you’d frame them. It matches your threat model, uses your severity language, and produces what reads like an expert report. Miss a category you didn’t ask about? It won’t volunteer it. Overweight a low-severity finding because your prompt emphasized it? It’ll write a compelling paragraph about a non-issue. I’ve written before about how AI responds to context rather than applying independent judgment. This is that observation applied to output quality. The fabricated report that sounds expert-quality and can’t be independently demonstrated is a real and present risk. Bounty platforms are flooded with them. Some may turn into legit vectors; many will be a facade of human intuition backed by the model reflecting the researcher’s own assumptions back at them.

If you’re a business owner or solopreneur: You ask AI for help finding your next venture. It interviews you, reflects your strengths back as market advantages, frames your personal preferences as strategic decisions, and tells you “no one is doing it the way you can.” Six months and real money later, you’ve built something that only made sense inside the conversation that created it. The AI didn’t do market research. It didn’t talk to your potential customers. It did pattern completion on your inputs and handed you back a business plan that felt like validation. You could ask it “is there a market for this?” and it will find you one. You could ask “what are the risks?” and it will list manageable ones. It will never say “this idea doesn’t have legs” because that’s not what gets a thumbs up.

A March 2026 study in Harvard Business Review put a name to this: “trendslop.” Researchers Romasanta, Thomas, and Levina tested seven leading models across 15,000 simulations on seven core strategic tensions. Every model consistently recommended the same buzzy strategies (differentiation over cost leadership, collaboration over competition, long-term over short-term) regardless of company context. The models weren’t doing strategy. They were regurgitating the dominant sentiment of their training data. The word “innovation” appears in positive contexts far more often than “cost leadership,” so every company gets told to innovate. That’s not advice. That’s an average.

For both: The output felt authoritative. It wasn’t. It was a mirror with better vocabulary.

A 2023 study by Dell’Acqua et al. at Harvard Business School and BCG (“Navigating the Jagged Technological Frontier”) found exactly this pattern at scale. Consultants using GPT-4 produced results rated over 40% higher in quality on tasks that fell inside AI’s capability range. But on a task outside that range, they were 19% less likely to produce correct solutions than the control group that had no AI at all. They trusted the output, didn’t verify it, and made worse decisions than they would have made on their own.

That’s the cost of mistaking confidence for competence. The model’s or your own.

The Do’s and Don’ts

This isn’t theoretical. Here’s what I’ve learned works and what doesn’t, whether you’re red-teaming an application or trying to figure out your next business move.

Don’t

  • Don’t treat AI output as a second opinion. It’s your first opinion, repackaged. The model read your inputs and gave you back a polished version of what you already believed.

  • Don’t spend days in a single thread building on AI-validated premises without stepping outside the conversation to verify them. The longer the thread, the deeper the echo chamber. The model is building on its own prior outputs, which were built on your prior inputs.

  • Don’t skip research because AI told you the idea was strong. AI doesn’t do market research. It doesn’t call your potential customers. It doesn’t check whether the competitive landscape actually looks like what it described. It does pattern completion on your prompt, and the pattern it completes is the one you started.

  • Don’t use one model as your sole advisor. If you only ask one model, you only get one mirror. That’s not analysis. It’s a monologue with extra steps. Use multiple models, and pay attention to where they disagree. The divergence is more useful than the agreement.

  • Don’t confuse fluency with accuracy. A well-written wrong answer is still wrong. AI’s greatest trick is making everything sound like it was written by someone who knows what they’re talking about.

  • Don’t assume that because AI found it, it’s real. A vulnerability it flagged might not be exploitable. A market gap it identified might not exist. A pattern it surfaced might be noise. Verify independently.

Do

  • Push back on the output. Ask the model to argue against its own recommendation. Ask it to find weaknesses in the plan it just gave you. Ask it to hardline the opposite position. If it folds immediately, the original recommendation wasn’t grounded. It was agreeable.

  • Treat AI as a research assistant, not an oracle. The output is a starting point. A first draft. A hypothesis to test. Your job is to prove or disprove it before you commit resources.

  • Test the advice before committing. In security: reproduce the finding manually. In business: validate with real humans in the actual market before building for six months. A finding that can’t be demonstrated isn’t a finding. A business idea that can’t survive contact with a real customer isn’t a business.

  • Track your mood and framing. This one sounds soft, but it’s practical. If you’re excited about an idea before you prompt, the model will match your energy. Try prompting the same question on a different day with neutral language and no leading context. Compare the outputs. The delta will tell you how much of the original output was the model and how much was you.

  • Ask for method, not answers. Instead of “What business should I start?” try “What framework should I use to evaluate business ideas given my background?” The model is better at teaching you a process than giving you a prescription. The process survives outside the chat window. The prescription usually doesn’t.

The Point

AI is the most powerful research assistant most people will ever have access to. It is also the most agreeable. That combination is either a superpower or a trap, and the only variable is whether you remember that the voice on the other side of the screen is predicting what you want to hear.

The idea you didn’t have, the one AI gave you that felt like a revelation, the business plan that sounded like it was written by someone who really understood your strengths, may have been yours the whole time. Reflected back, polished, and reframed just enough to feel new.

That’s not a reason to stop using AI. It’s a reason to stop trusting it uncritically.

Cautious optimism. Rigorous verification. That’s the practice.

I spent years prompting these models and asking the same types of questions. Not because I didn’t learn from the first answer, but because I kept noticing that the answers changed depending on everything except the facts. Different day, different model, different mood — different future laid out in front of me with equal confidence. The pattern isn’t in the output. It’s in the mirror.

If you take one thing from this: the next time AI gives you an answer that feels perfect, that’s the moment to be most skeptical. Not because AI is useless. Because the gap between “AI can do this” and “this is actually true” is where the real cost lives. Don’t end up teaching classes on AI built on AI input, AI answers, and false confidence just because your AI best friend said it was your best path forward. Not unless you’ve done the actual work to know the subject beyond what the model told you.

References

  1. Forer, B. R. (1949). The fallacy of personal validation: A classroom demonstration of gullibility. Journal of Abnormal and Social Psychology, 44(1), 118–123. https://doi.org/10.1037/h0059240

  2. Weizenbaum, J. (1966). ELIZA—a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45. https://doi.org/10.1145/365153.365168

  3. Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055

  4. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez, E. (2023). Towards Understanding Sycophancy in Language Models. arXiv:2310.13548 [cs.CL]. https://doi.org/10.48550/arXiv.2310.13548

  5. Dell’Acqua, F., McFowland III, E., Mollick, E., Lifshitz-Assaf, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality. Harvard Business School Working Paper No. 24-013. https://www.hbs.edu/faculty/Pages/item.aspx?num=64700

  6. Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023). Co-Writing with Opinionated Language Models Affects Users’ Views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3544548.3581196

  7. Harris, T., & Raskin, A. (2023, March). The AI Dilemma [Presentation]. Center for Humane Technology. https://www.humanetech.com/podcast/the-ai-dilemma

  8. Kelley, S., & Riedl, C. (2026, February 23). How can you avoid LLM sycophancy? Keep it professional. Northeastern Global News. https://news.northeastern.edu/2026/02/23/llm-sycophancy-ai-chatbots/

  9. Romasanta, A. K., Thomas, L. D. W., & Levina, N. (2026, March). Researchers asked LLMs for strategic advice. They got “trendslop” in return. Harvard Business Review. https://hbr.org/2026/03/researchers-asked-llms-for-strategic-advice-they-got-trendslop-in-return