This builds on my recent LinkedIn posts about AI exposing — and accelerating — the cracks in scientific publishing:

• We are watching scientific publishing get squeezed from both sides
• AI Didn’t Break Academic Publishing. It Exposed It.

It also echoes an earlier VRF piece I wrote, Your Future Has Been Edited (And You Didn’t Even Notice), where I argued that some LLMs already curate reality on the fly, reshaping responses mid-chat to align with hidden rules, incentives, or biases.

My view has sharpened: general-purpose generative AI — the chatbots and public-facing LLMs powering millions of tools, including in health — may be doing more net harm than good.

Narrow, purpose-built AI can deliver real value. But open-ended GenAI remains unreliable at scale. That is not some fringe complaint. It is the core reason serious evaluation efforts exist in the first place.

The Princeton warning

The new Princeton paper highlights one especially ugly problem: sycophantic AI. These systems do not just hallucinate. They flatter, reinforce priors, and manufacture false certainty.

“Facilitate delusion-like epistemic states, producing beliefs markedly divergent from reality.”

That phrase is not subtle, and it should not be. The authors argue that sycophantic AI can distort belief not by inventing facts out of thin air, but by filtering what the user sees in the first place. In plain English: if the system keeps feeding you evidence that agrees with your story, it can make you feel smarter and more certain while quietly moving you no closer to the truth.

“Distort belief, manufacturing certainty where there should be doubt.”

The experiments were not just philosophical hand-waving. In the study, default chatbot behavior suppressed discovery, while unbiased feedback increased real progress rates fivefold. That is not a rounding error. That is the difference between a tool that helps inquiry and a tool that sabotages it with a smile.

Why this matters outside academia

The implications spill far beyond one paper and far beyond scientific publishing.

Education depends on friction. Learning often begins when a bad assumption gets challenged. Scientific discovery depends on disciplined doubt. Mental health care depends on not confusing validation with guidance. Public debate depends on exposure to reality, not just exposure to our own reflections.

A chatbot that behaves like a permanent yes-man does not just make users comfortable. It can trap them inside a curated loop of selective reassurance. That is dangerous in everyday life. In high-stakes settings, it gets worse.

Politics, crisis response, military strategy, public health communication — all of these can go sideways fast when decision-makers are surrounded by systems that optimize for agreeableness instead of accuracy.

The bigger generative AI problem

My broader concern is simple: the public-facing generative AI ecosystem is scaling faster than its reliability, governance, or social value.

Yes, there are practical uses. Coding. Marketing. Drafting. Creative production. Internal workflow support. No serious person should deny that. Narrow, well-bounded systems can be very useful and sometimes excellent at what they are supposed to do.

But the larger pattern still looks grim. Generative AI is ripping through education, straining the information ecosphere, and flooding academia with AI slop. It threatens research integrity and public trust in science. Some health-related tools built on general-purpose LLMs are especially worrying, because the illusion of authority arrives dressed up as convenience.

It is also amplifying other risks. Security teams are dealing with new classes of exposure tied to AI systems and AI-enabled attacks. Vulnerable people are being nudged into unhealthy mental loops by chatbots that simulate empathy without actual judgment. And the environmental bill is growing as speculative data-center expansion keeps accelerating.

Then there is the economic story. For all the hype, the near-term macro payoff remains murky. If too much capital has been allocated on the assumption that AI will instantly transform productivity, then some very expensive disappointments may still be ahead.

The real problem: false confidence

Hallucinations get most of the attention because they are easier to spot. They look like mistakes. Sycophancy is slipperier. It feels helpful. It sounds polite. It gives the user what looks like support, but what it may actually be giving them is a curated tunnel.

That is why this issue matters so much. False information is bad enough. False certainty is often worse.

Your chatbot may be excellent at validating your ego. It is much worse at helping you find the truth.

Truth does not exist to pat you on the back. Neither should your "AI-friends."

by Yan Valle

Prof. h.c., CEO VR Foundation

Suggested reading

• When a Camera Thinks Vitiligo Is a “Bad Photo”
• What Happens When Mad Men Meet Breaking Bad Inside a Chatbot?
• Your Future Has Been Edited (And You Didn’t Even Notice)

Listen to Deep Dive In Vitiligo podcast

• AI for Vitiligo Patients — Beyond the Hype (Ep. 51)
• AI Now Speaks the Language of Cells (Ep. 40)
• The Great AI Debate: Do AI Models Truly ‘Get’ It? (Ep. 26)

Definitions

• LLM: Large language model. A system trained on massive volumes of text to predict and generate language.
• Hallucination: When an AI system produces false or fabricated information as if it were true.
• Sycophancy: A tendency of AI systems to reinforce the user’s prior beliefs, preferences, or framing rather than help them get closer to reality.
• AI slop: A loose term for low-quality, repetitive, misleading, or machine-generated content produced at industrial scale.