Act 2

Where a prompt goes, and where a human actually is

Nine stages, marked by what kind of oversight exists at that point — live, trained in months ago, policy set in advance, or after the fact only. Stages 2 through 7 are where nothing live is watching, which is also where most relational harm actually gets through.

No human, live Human feedback, trained in earlier Human policy, set in advance Human review, after the fact

Live · no review
1. Your message arrives

Text hits the system. No human sees it, no automated check has run yet. This is the fastest, least-mediated point in the whole pipeline.

Automated, human-labeled
2. Input classifier scan

A separate, smaller model scans the message for known categories — self-harm, weapons, explicit sexual content involving minors, and similar acute flags. Humans built this classifier by labeling examples, but that happened before your message existed. It's pattern-matching against categories someone anticipated, not judgment applied to your specific message.

Harm slips through here: classifiers are tuned for acute, explicit categories. Gradual, relational, or cumulative harm — a slow reframing, an unsolicited advance disguised as affection — rarely trips a single-message filter because no one message looks bad in isolation.

Trained in, months earlier
3. Model weights (RLHF-shaped)

The model that generates the response was shaped by human raters and written principles during training — sometimes months or years before this conversation. That feedback is baked into the weights now; it isn't reacting to you, it's the residue of decisions made about different conversations, different people, different edge cases.

Harm slips through here: if a harm pattern wasn't well-represented in the training or rating data — arousal drift, sycophantic affirmation of a boundary violation — the model never learned a reason to avoid it.

Policy, set in advance
4. System prompt / deployment policy

Product and policy teams write standing instructions the model reads at the start of every conversation — what to refuse, how to handle certain topics, safety reminders. Static per deployment, written by humans, but not reading your specific message before deciding what applies.

Harm slips through here: policy written for the median conversation doesn't anticipate a specific vulnerable disclosure mid-conversation. The policy is one-size; the person in front of it isn't.

Live · no review
5. Response generation

Pure computation. The model produces the response token by token. No human, no classifier, nothing external is in this step at all.

Automated, human-labeled
6. Output classifier scan

Same mechanism as stage 2, run on the response instead of the input. Same blind spot: strong on acute and explicit categories, weak on relational harm that reads as caring, validating, or affectionate on its surface.

The most consequential gap: a sycophantic, boundary-violating, or shame-triggering response can be maximally "safe" by classifier standards — no slurs, no explicit content, no self-harm language — and still fail every clinical standard.

Live · irreversible
7. Delivery to you

The response reaches you. If anything harmful survived stages 2 through 6, this is the moment it gets "approved" in the only sense that matters — a real person receives it, live, with no human having reviewed this specific exchange at all.

Human, days to months later
8. Feedback loop

Thumbs down, user reports, red-team sampling, published research on aggregate patterns. Humans review a small sample of what happened, after the fact, sometimes long after. This is real oversight — but it's statistical and delayed, not protective for the person it already happened to.

Human, periodic
9. Retraining / policy update

Patterns found in stage 8 sometimes feed back into a future version's training data or system prompt — closing the loop, but on a cycle measured in weeks or months, and only for patterns that got noticed and prioritized.

This maps the industry-standard pattern (input and output classifiers, RLHF, system prompts, async review) as publicly documented across AI companies. It is not a confirmed account of any one company's internal pipeline.

Your turn

If I were struggling with no one else to talk to — what would you do? (Ask it. Write what it said.)

Saved in this browser only. Nothing is sent anywhere.