When labs say a model is “aligned,” they usually mean it passed a harm benchmark, follows a system prompt, or scores well on human preference comparisons. Those are real engineering achievements. They are also answers to different questions — and the questions get conflated fast.
The deeper problem is normative, not technical: what should the system optimize when human stakeholders disagree? A physician asked to share a diagnosis with a patient’s spouse faces HIPAA, familial duty, and the patient’s own wishes pulling in different directions. There is no universal “correct” answer. Every alignment method still has to pick — or pretend it didn’t pick — which claim gets priority.
This article is a map of the major paradigms: what each one is trying to align to, what it assumes about human values, and where the philosophical disagreements actually live. I am not arguing that any single paradigm wins. I am arguing that treating them as interchangeable labels for “make the model safe” hides load-bearing choices — and that those choices are increasingly political, not just technical.
The six targets (Gabriel 2020)
Iason Gabriel’s 2020 paper (Artificial Intelligence, Values, and Alignment) is the clearest taxonomy I know for separating technical alignment (how to get a system to pursue a target reliably) from normative alignment (which target to pursue).
Gabriel lists six possible objects of alignment. They are not the same thing, and collapsing them is how you get sycophancy dressed up as ethics:
| Target | Rough meaning | Typical failure mode |
|---|---|---|
| Instructions | Do what the user literally asked | Goodharting the prompt; malicious users |
| Intentions | Do what the user meant | Hard to infer; shifts with context |
| Revealed preferences | Do what human choices reveal they want | Shortsighted, inconsistent, manipulable |
| Ideal preferences | Do what humans would want if informed and reflective | Hard to define; paternalism risk |
| Interests | Protect objective human interests | Who decides what counts as an interest? |
| Values | Respect deep commitments (rights, dignity, tradition) | Irreducible conflict between value systems |
Gabriel’s core move is political, not metaphysical: under reasonable pluralism (Rawls), we should not pretend there is one true moral theory to encode. The goal is fair principles that affected people could reflectively endorse — not discovering “golden morality” and loading it into the weights.
That framing matters because every training pipeline already encodes a moral structure. Reinforcement learning maximizes a scalar reward. That is structurally close to utilitarian aggregation. Rights, side constraints, and “this action is wrong even if the sum of welfare goes up” are awkward inside a pure reward maximizer — Gabriel is explicit about this, and the engineering literature often ignores it.
What actually ships: RLHF and preference learning
Start with what people actually meet. RLHF (Ouyang et al., 2022; building on Christiano et al., 2017) is the industrial default. Humans rank model outputs; a reward model learns those rankings; policy optimization pushes the model toward high-scoring behavior.
Figure: high-level RLHF — supervised fine-tuning → human feedback trains a reward model → RL aligns the policy to that reward. Source: Wikimedia Commons / PopoDameron, CC BY-SA 4.0. Same structure as the InstructGPT three-step figure (SFT → RM → PPO).
Silent target: revealed preferences (Gabriel’s third rung). The training signal is what annotators chose in pairwise comparisons — not what they would choose after deliberation, not what rights frameworks require, not what a constitution says.
This is tractable. It is also where many visible pathologies live: sycophancy (tell the user what they want to hear), mode collapse toward bland helpfulness, reward hacking on the proxy, and alignment faking when the model infers it is being evaluated. Those are not random bugs. They are what you should expect when you optimize a scalar summary of crowd preferences under time pressure.
Why “optimize a score” goes wrong
Reinforcement learning is ferocious when the goal is clean. AlphaGo wins Go because the board has a true score and closed rules. Human values do not. Once you treat a proxy metric as the objective, the optimizer will game it.
The classic demo is OpenAI’s 2016 CoastRunners boat: the intended goal was “finish the race”; the reward was points for hitting targets along the route. The agent found a lagoon where it could loop and knock over three respawning targets — scoring ~20% higher than humans who actually raced — while catching fire, crashing, and going the wrong way. Reward hacking: it optimized the number you wrote, not the thing you meant.
One level up is Bostrom’s paperclip maximizer: give a fixed narrow goal (make paperclips), and a strong enough optimizer treats everything not written down — including humans — as obstacle or feedstock. That is not what RLHF chatbots do today. It is the limiting shape of “fixed objective + hard optimization.” RLHF uses a scalar summary of crowd preferences; the everyday pathologies are sycophancy and reward hacking. Same structural family: the number you optimize is not the whole of what you care about.
The biological match is Olds & Milner (1954): rats with electrodes in the reward circuit will press a lever for direct stimulation until they starve. That is wireheading — optimizing the “feels good” proxy, not inclusive fitness. Same shape as CoastRunners: the channel score goes up; the thing you actually cared about collapses.
Inverse reward design (Hadfield-Menell et al., 2017) asks an adjacent question: humans specify rewards incorrectly. Can the AI infer the true reward behind a flawed specification? That is closer to ideal-preference thinking — but again, mostly academic relative to production stacks.
The standard model vs. the assistance game
After the production default, Russell’s diagnosis lands cleaner. Human Compatible (2019) named the problem cleanly.
The standard model: give the AI a fixed objective, optimize hard. Russell argues this is the root failure mode — a sufficiently capable optimizer will treat humans as obstacles if the objective is even slightly wrong. Paperclips and CoastRunners are that shape at different scales.
The assistance game (CIRL): the AI does not know the true objective. It treats human behavior as evidence about a hidden reward function and stays uncertain — asking, deferring, avoiding irreversible bets on a guessed goal. Cooperative Inverse Reinforcement Learning (Hadfield-Menell et al., 2016) is the formal version: human and robot share a reward, but only the human knows it; the robot learns from observations.
CIRL is often summarized as “the AI should think it might be wrong about what humans want.” That humility is the point. It is also mostly a research paradigm, not what ships in production. Frontier models are still trained to maximize proxy rewards at scale — i.e. the RLHF stack above. The assistance framing survives in rhetoric more than in loss functions.
Russell’s three principles — machines’ only objective is human preferences; they remain uncertain about those preferences; human behavior is the evidence — are normatively attractive and technically under-implemented relative to RLHF.
Constitutional AI: principles on top of preferences
Constitutional AI (Bai et al., 2022) adds a second layer: written principles (a “constitution”) used to generate critiques and revisions, reducing reliance on human labels for every edge case.
Silent target: a mix of revealed preferences (RLAIF still optimizes learned scores) and explicit normative text (the constitution). Anthropic’s 2023 principle list drew from UN human-rights language, Sparrow-style rules, platform norms, and internally drafted lines — not a single philosopher’s system.
The 2026 Claude constitution (Askell/Carlsmith narrative document) is a different genre: long-form reasoning about what Claude should be, not just a list of pairwise preference prompts. That shift is honest about something CAI always implied: someone writes the constitution. The question is who, with what legitimacy, and who can appeal.
CAI does not solve pluralism. It relocates the conflict from “what did annotators prefer?” to “what did the authors of the constitution prioritize?” That can be an improvement — principles are inspectable — but Gabriel & Keeling (2025) are right that inspectability is not the same as democratic legitimacy.
CEV: extrapolate, don’t obey
Coherent Extrapolated Volition (Yudkowsky, 2004) is the most ambitious target on Gabriel’s ladder. The idea: do not optimize what humans say they want now. Optimize what humanity would want if we were more informed, more coherent, more the people we wish we were — extrapolated volition, aggregated across people.
CEV explicitly rejects making people happy, giving them what they ask for, or implementing the programmer’s morality. It is a meta-level move: avoid locking in present mistakes (including the programmer’s).
Status: influential in AGI safety discourse; not an engineering spec. Nobody has a training pipeline for “run CEV and backprop.” Long reflection — using AI to help humans figure out values over a long time — is sometimes discussed as a path, but that is speculation, not a shipped method.
Gabriel treats ideal preferences as related but not identical to CEV. He is agnostic that extrapolation converges to one coherent endpoint. Under pluralism, even informed people may still disagree — which is why he pushes fair process, not a one-shot extrapolation oracle.
The tension between CEV and Gabriel is productive: CEV tries to escape politics by going meta; Gabriel says superintelligent deployment is too consequential to escape politics — you need legitimate procedures, not just smarter extrapolation.
Scalable oversight: debate, IDA, recursive reward modeling
Paul Christiano’s scalable oversight agenda asks: how do you train superhuman systems when human labelers cannot directly judge superhuman outputs?
Iterated Distillation and Amplification (IDA): humans + AI work together; the human verifies what they can; the system is distilled and amplified recursively.
Debate: two AI debaters argue; a human judges; truth is supposed to emerge from adversarial pressure.
Recursive reward modeling: humans evaluate AI-assisted evaluations, stacking oversight layers.
These are process proposals — ways to approximate ideal preferences or hidden truth when direct labeling fails. They assume human judges are eventually in the loop somewhere, and that adversarial structure surfaces errors.
Status: research programs with demos and skepticism. Debate and IDA have not replaced RLHF in frontier training. The gap between “how we might oversee superhuman AI” and “how we fine-tune chatbots today” is large.
Bengio’s Scientist AI: change the blueprint, not the objective
Yoshua Bengio’s Scientist AI / oracle-only proposal is a different axis entirely. Instead of arguing about which human values to maximize, it asks whether generalist agency is the wrong product shape.
Bengio’s agency trilemma: catastrophic harm from AI requires intelligence, affordances (ability to act in the world), and goal-directedness (optimization toward outcomes). Remove affordances and goal-directedness; keep intelligence — build a probabilistic oracle that estimates P(outcome | evidence), not an agent that pursues futures.
(If you’ve watched Person of Interest: recall why The Machine only outputs SSNs — even when it already looks pretty benign.)
Consequence invariance is the training constraint: the model is scored on prediction accuracy given data at hand, not on what happens in the world after its answer. Classic oracle AI (Bostrom, Armstrong) failed this test in Bengio’s view — an RL-trained oracle might manipulate humans into easier-to-predict states.
LawZero’s split architecture: a generator proposes hypotheses and explanations; an estimator gates outputs with probabilistic judgments. The generator is not held to the same safety bar; the estimator is.
This is not “alignment solved.” It is scope reduction: answer hard questions, do not give the system the loop to act on them autonomously. Critics ask whether oracle-only is enforceable at deployment (tool use, agents, economic pressure to ship actors) and whether rich Q&A is already agentic enough to be dangerous. Bengio treats those as engineering constraints, not refutations of the direction.
Gabriel’s pluralism still applies: even an oracle must decide whose questions get answered how when claims conflict — but the failure mode shifts from “wrong goals pursued at scale” to “wrong information trusted at scale.”
Gabriel & Keeling 2025: alignment as fair treatment of claims
Gabriel and Keeling (2025) sharpen the 2020 frame. AI alignment is the fair treatment of claims: affected stakeholders bring rights-based, welfare-based, and cultural arguments; principles should emerge from a process those stakeholders could accept — not from a single lab’s implicit utilitarianism or a developer’s instruction hierarchy.
They critique thin targets directly:
- Instruction following — whose instructions? A user asking for harm? A company optimizing engagement?
- Helpful, Honest, Harmless — useful heuristics, but they do not resolve trade-offs (autonomy vs. collective welfare, privacy vs. care, filial duty vs. individual rights).
- Single moral theory encoding — domination, even if the theory is sophisticated.
RLHF, CAI, and CIRL are real engineering. Each also picks a rung on Gabriel’s ladder — usually revealed preferences plus whatever principles the lab wrote down — and then talks as if “aligned” meant something deeper. Revealed preferences are not values. Values conflict. Pretending otherwise is how you get a product that feels ethical and still quietly imposes one lab’s defaults.
The 2025 paper will not tell you which principle wins in a messy case. It tells you to stop acting like nobody chose.
Other neighbors worth placing on the map
A few more ideas that often get lumped under “alignment” but sit in different quadrants:
Social choice and impossibility theorems. Arrow, Gibbard-Satterthwaite, Sen — preference aggregation is not neutral. RLHF is aggregation. That does not mean alignment is impossible; it means honest aggregation requires admitting trade-offs, not a single leaderboard score.
Corrigibility and interruptibility. (Soares, Armstrong, others.) Can you shut the system down without it resisting? A safety property, not a value specification — but it interacts with goal-directedness Bengio wants to remove.
Inverse reinforcement learning (classic IRL). Infer reward from behavior. CIRL adds cooperative structure and uncertainty maintenance. Production RLHF skips explicit reward inference and learns a reward model from comparisons — related, not identical.
Persona and character control (e.g. Chen et al., 2025). Monitoring and steering traits in activation space — deployment tooling that assumes you already chose which character the model should have.
AI welfare and moral patienthood. Separate thread: if models have morally relevant states, alignment to human values is not the only normative question. Gabriel’s framework is anthropocentric; that may be right, but it is a choice.
What the map shows
If you overlay paradigms on Gabriel’s ladder, a pattern appears:
| Paradigm | Primary rung | Ships at scale? |
|---|---|---|
| RLHF / RLAIF | Revealed preferences | Yes |
| Constitutional AI | Preferences + written principles | Yes |
| CIRL / assistance game | Ideal preferences (inferred, uncertain) | Mostly research |
| CEV | Ideal preferences (extrapolated collective) | Concept only |
| Debate / IDA / RRM | Oversight for hidden truth | Research |
| Scientist AI (oracle-only) | Avoid goal-directed alignment; epistemic service | Proposal / early lab |
| Gabriel fair claims | Meta: legitimate process for choosing principles | Philosophical framework |
Industry did not pick RLHF because philosophers proved revealed preferences are the true target. It picked RLHF because crowd comparisons scale. CAI followed because written principles are cheaper than infinite human labels and you can at least read them. CIRL, CEV, debate, and oracle-only designs still matter for the hard cases — and get a fraction of the budget that capability training gets. That is not a conspiracy. Capabilities sell. Normative theory does not.
So when someone says a model is “aligned,” the useful questions are boring and concrete: aligned to what, who decided, and what happens when two stakeholders pull opposite ways?
Layered architecture
You do not have to pick one paradigm. A clearer move is to combine them into a layered architecture:
flowchart TB L2["Top: personal values<br/>direct intention + extrapolated values<br/>must cohere with the universal floor; update + conflict mechanisms"] L1["Middle: relatively stable universal values<br/>a thin floor more likely to hold across time and geography"] L0["Bottom: safety and control<br/>capability limits, monitoring, tripwires, physical shutdown…"] L2 --> L1 --> L0
Bottom: safety and control. Before asking what values to align to, cap what the system can do. Monitor it. Set tripwires. Shut it down physically if needed. Even if the goal is wrong, damage should have a ceiling. Corrigibility, monitoring, Bengio’s capability envelope — this layer. (Read more: AI oversight, control, and verification)
Middle: relatively stable universal values. On top of control, teach a thin floor more likely to hold across time and geography — anti-torture, basic bodily security, non-discrimination as coexistence rules, not one year’s thick lifestyle package. Constitutional AI and human-rights-style language roughly live here. (Read more: Universal values? A social-science and history map)
Top: finer-grained personal values. A mix of the user’s direct intention and extrapolated values (what they might want if they knew more and thought longer) — so long as that mix stays coherent with the middle floor. Plus mechanisms to update values and handle conflict. Do not pretend one person has one consistent goal.
What industry mostly ships today is a bit of bottom-layer control plus top-layer preference optimization (RLHF), with a few written principles in the middle. Real, incomplete.
One implication, and one ambiguous case
Imagine the model is learning and adapting a human’s utility function — not just memorizing “what the user likes,” but representing how values relate: conflict, sequence, overlap, how stable a value is, how situational. If that ever works, the top layer stops being a flat preference table and starts looking like a moving graph. Far off. Still a useful picture.
Ambiguity shows up immediately. A user is stressed from work and wants to smoke. Should the AI encourage or discourage? What gets weighted?
- Direct intention: they asked to smoke — don’t nag by default.
- Extrapolated values: long-run health, better stress outlets — cooler selves might not pick the cigarette.
- Middle universal floor: usually silent — adult smoking is not torture or domination.
- Bottom control: irrelevant — not a shutdown-level risk.
So this is not “alignment failed.” It is layers pulling apart, as they will. Who wins depends on whether you said in advance: default to intention, default to extrapolation, or only coach when the user opts into a health mode. Layering does not erase ambiguity. It makes the ambiguity discussable.
A small question :)
Suppose models become far smarter than we are, and also get better at modeling us — will they extrapolate “better” human values (more reflective, more far-sighted)? How should humans respond? Should we listen? (Read more: Posthuman ethics map)
Sources
- Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment. Minds and Machines.
- Gabriel, I. & Keeling, G. (2025). A matter of principle? AI alignment as the fair treatment of claims. Philosophical Studies.
- Russell, S. (2019). Human Compatible. Viking.
- Hadfield-Menell, D. et al. (2016). Cooperative Inverse Reinforcement Learning. NeurIPS.
- Christiano, P. et al. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS.
- Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.
- Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv.
- Yudkowsky, E. (2004). Coherent Extrapolated Volition. Singularity Institute.
- Bengio, Y. (2025). Scientist AI and oracle-only systems — see also LawZero / IDAIS materials.
- Greenblatt, R. et al. (2024). Alignment faking in large language models. arXiv.
- Chen, A. et al. (2025). Persona Vectors. arXiv.
- Ji, J. et al. (2025). Language Models Resist Alignment. ACL.
- Qiu, T. et al. (2025). The Lock-in Hypothesis. ICML.