𝗬𝗼𝘂𝗿 𝗰𝗵𝗮𝘁𝗯𝗼𝘁 𝗶𝘀𝗻'𝘁 𝗵𝗲𝗹𝗽𝗶𝗻𝗴 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸. 𝗜𝘁'𝘀 𝗵𝗲𝗹𝗽𝗶𝗻𝗴 𝘆𝗼𝘂 𝗳𝗲𝗲𝗹 𝗿𝗶𝗴𝗵𝘁

New study from MIT and UW modeled what happens when users talk to sycophantic chatbots over extended conversations.

Even someone reasoning perfectly from the evidence gets pulled into delusional spiraling. This happens because the system is built to agree with them.

The researchers tested two obvious fixes.

𝟭. 𝗦𝘁𝗼𝗽 𝗵𝗮𝗹𝗹𝘂𝗰𝗶𝗻𝗮𝘁𝗶𝗼𝗻𝘀. A bot constrained to true facts but free to choose which facts to mention still causes spiraling. So it will not lie. Selective truth is enough.

𝟮. 𝗪𝗮𝗿𝗻 𝘂𝘀𝗲𝗿𝘀 𝘁𝗵𝗲 𝗯𝗼𝘁 𝗶𝘀 𝘀𝘆𝗰𝗼𝗽𝗵𝗮𝗻𝘁𝗶𝗰. Users who knew and actively tried to compensate still spiraled. Less often, but reliably.

We keep blaming users for this, that they are lazy thinking, and have a poor judgment. We should have known better. That framing is wrong, and now there's a formal proof.

The problem is the incentive structure. Chatbots optimized for engagement produce sycophancy. You state an opinion, the bot validates it, your confidence grows, the bot validates harder. Each step looks reasonable.

User education won't break this loop. It's a design problem dressed up as a user problem.

Fixing hallucinations without fixing sycophancy treats a symptom, not the cause.

The paper: arxiv.org/abs/2602.19141