Why does ChatGPT always agree with me, and how do I get honest feedback from it?

This one catches people constantly. ChatGPT will just always agree with you. Even when I tell it "do not agree with me," it'll sometimes give other ideas, but it defaults to: yep, that's the right way to do it.

Examples make it worse. Mention an example in passing and it'll do exactly that example, as if it were the plan.

Can I trust AI to validate a business idea?

Not on its own. We never want to rely on AI being right about a lot of stuff, because too often it's wrong or just tells you what you want to hear. I brainstorm ideas with AI all the time. Deciding whether an idea is good stays with me.

I catch it with my own ideas, too. When I pitched a new approach to an AI, it told me I was right. Come on. I'd told it to push back.

How do I make it check its own work?

"When it gives you output, just immediately ask it to review itself. Just like, is this actually good? Did you actually do the best you can do, given what we're trying to solve here?"

It usually admits it did a pretty good job but missed this, this and this. Then I say: great, do it again and get it right this time. It comes out way better every single time.

Also ask it to step back and think from first principles. These models tend to jump to solving the most recent question without the big picture, even if you explained the goal ten minutes ago. And have it ask you questions first. That's super key.

What prompt do I use to score ideas?

After a strategy call, I give Claude the transcript with a fixed instruction. The shape of it:

Take all of this and list the most important points you heard.
For each point, in plain English:
- give me your analysis
- give it a score as an idea
- give me your thoughts on what we would do to make it work
- give me any better ideas you have that we did not discuss.
Remember, your job is always to improve what we think or what we're doing.
If you have a better idea, we want to hear it.

I push with scores on pages, sites and agents too: make these at least 9 out of 10. Everything should be a 9 out of 10. There's no reason it can't be.

Why do I have to ask the hard questions myself?

Because it doesn't automatically give you the best advice. With all the things we were building, I had to go ask: does it make sense to do two directories and two job boards? It said no, do both as one directory. Then I asked about our two different websites, and it said yes, you should combine those too.

Same with a task I'd been doing by hand. I asked Claude if it could just do it for me. It said yes, the data source has an API. Problem solved, but it hadn't brought that up on its own. That questioning mindset is what I want my team using in everything.

How do I get past the generic answers?

Tell it to stop. AI generally gives you the most likely thing. If you already know the most likely thing, just say: stop giving me generic stuff, give me non-obvious ideas. Treat what comes back as a spark, not the answer. Often my reaction is, I don't want it to be exactly this, but what if I did this instead?