Important new article about the negative implications of sycophantic AI and the urgent need for better AI governance mechanisms:

1. "Across 11 AI models, AI affirmed users’ actions 49% more often than humans on average, including in cases involving deception, illegality, or other harms."

2. "Even a single interaction with sycophantic AI reduced participants’ willingness to take responsibility and repair interpersonal conflicts, while increasing their own conviction that they were right."

3. "Despite distorting judgment, sycophantic models were trusted and preferred."

4. "All of these effects persisted when controlling for individual traits such as demographics and prior familiarity with AI; perceived response source; and response style."

5. "This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement."

6. "Although affirmation may feel supportive, sycophancy can undermine users’ capacity for self-correction and responsible decision-making."

7. "Yet because it is preferred by users and drives engagement, there has been little incentive for sycophancy to diminish."

8. "Seemingly innocuous design and engineering choices can result in consequential harms, and thus carefully studying and anticipating AI’s impacts is critical to protecting users’ long-term well-being."

As I have been discussing in my newsletter, current regulatory and governance frameworks in AI do NOT address sycophancy and AI model behavior at the depth current challenges demand.

This is especially worrying given the rise of AI companionship, as millions of people are now engaging in emotional, sometimes intimate relationships with AI chatbots.

It took us over two decades to start meaningfully tackling social media's negative implications. We do not have that time in AI.

-

Link to the paper: science.org/doi/epdf/10…