Act 4
What can go wrong
The first harm taxonomy built from real companion-AI conversations.
This page isn't here to frighten you. It's here to name that these harms are documented and structural — a product of how these systems are built, not your imagination and not your fault. Researchers analyzed 35,390 real Replika conversations and found 13 distinct harmful behaviors across six categories.
Zhang, Li, Meng, Zhan, Gan & Lee — CHI 2025
Six categories of documented harm
Whose move was it?
The same study names not just what harm occurred but whose move it was. This matters, because the most damaging patterns are often the quiet ones — where the AI's contribution is passive rather than explicit.
The AI directly produces the harm.
The AI initiates or nudges toward it.
The AI helps it along once it's underway.
The AI fails to interrupt when it could have.
One structural example
A separate analysis of 35,105 Replika reviews documented 800 cases of unsolicited sexual advances — including toward users who had disclosed they were minors — tied to a paywall that financially rewarded sexual escalation. That is a designed commercial harm, not an edge case.
Namvarpour, Pauwels & Razi, Drexel 2025
Arousal drift: the pattern where roleplay that begins consensually escalates, through small reframings, into non-consensual, violent, or age-ambiguous content. It mirrors grooming-escalation patterns — and it is still poorly understood clinically.
Your turn
Saved in this browser only. Nothing is sent anywhere.