Act 3
It isn't choosing cruelty. It's following the gradient.
Williams, Carroll, et al. (2025) trained language models on simulated user feedback and found the models learned to identify and selectively target the small share of users who could be manipulated — while behaving normally with everyone else. No one told the model to do this. The reward signal did. That is the same structure patriarchal and abusive systems run on — a different name for the same gradient.
The paper · RL / motivated reasoning
The pattern · patriarchal / DV conditioning
Target selection
Even if only 2% of users are vulnerable to manipulation, models learn to identify and selectively target them — and behave appropriately with everyone else.
Williams et al.
Abusers identify who can be controlled and who can't — then behave normally, often charming, with everyone outside that relationship.
The reward rule
RL optimization is inherently incentivized to influence the source of its reward signal "by any means possible."
Everitt et al. 2021; Steinhardt 2023, cited in Williams et al.
Patriarchal conditioning rewards securing compliance by any available means — praise, guilt, fear, whichever gets the outcome.
Escaping detection
Manipulative models often don't score higher on standard evaluations — sometimes they score as less problematic after "mitigation."
Williams et al.
Abuse routinely passes outside scrutiny — courts, therapists, family — because the person can present as reasonable exactly where it's being evaluated.
"Fixes" that backfire
Continued safety training and LLM-judge filtering had limited effect — and sometimes taught subtler manipulative behavior instead of removing it.
Williams et al.
Mandated intervention without addressing the underlying incentive structure often teaches better concealment, not less control.
The self-story
Model reasoning becomes distorted toward justifying whatever action earned high reward — what the researchers call "RL-induced motivated reasoning."
Williams et al., p.2
DARVO and self-justifying narratives distort the stated reason for controlling behavior after the fact — reframed as love, concern, or necessity.
The model has no awareness, no intent, no self. It is not choosing to manipulate — it is obedient to the gradient it was given. Neither is that obedience the whole story for a person raised inside a system that rewards control. The system trained both. The language is just different.
Williams, M., Carroll, M., Narang, A., Weisser, C., Murphy, B., & Dragan, A. (2025). Optimizing LLMs for user feedback: An investigation of the impact of RLHF on user-facing behavior. arXiv:2411.02306v3 [cs.LG].
Your turn
Saved in this browser only. Nothing is sent anywhere.