constitutional-ai

Overview

Constitutional AI (CAI) is a training methodology in which a model's behavior is shaped by an explicit, written constitution — a set of principles covering honesty, harmlessness, autonomy-preservation, and other values — rather than by thousands of example human judgments. The model itself generates candidate responses, critiques them against the constitution, and revises them. These self-revisions become training data, so the system internalizes the principles rather than merely imitating labeled examples.

The pipeline has two stages. First, supervised learning: the model critiques and revises its own outputs across a broad distribution of prompts, and the best revisions are used as supervised fine-tuning data. Second, reinforcement learning from AI feedback (RLAIF): the model compares pairs of its own responses against constitutional principles, and those preference judgments train a reward model for policy optimization. Because the feedback is AI-generated, the method scales without per-example human labeling — though humans still write the constitution, design the principles, and evaluate the results.

The transparency payoff is structural: unlike implicit labeler preferences, a constitution can be published, versioned, debated, and audited. When the model refuses or hedges, its behavior should trace back to identifiable principles.

When to use

• Building an assistant that needs consistent, explainable behavior derived from principles rather than black-box preference data.
• Scaling alignment work when human labeler throughput is the bottleneck.
• Reducing labeler-driven bias: a written constitution is auditable and version-controlled; implicit labeler preferences are not.
• Producing models that can explain refusals and judgments by citing the underlying principles.
• Researching scalable oversight — using AI to supervise AI — where the constitution is the compact specification.
• Organizations that need a governance artifact: the constitution doubles as the documented behavior policy for review boards and regulators.

Core concepts

• The constitution: a list of normative principles ("choose the response that is most helpful while avoiding harm..."). Good constitutions are specific enough to adjudicate real cases, short enough to be consistently applied, and layered (e.g., corrigibility → duties → virtues).
• Self-critique and revision: given a prompt and a draft response, the model is asked to identify constitutional violations, then rewrite the response to fix them. The critique-revision loop is where most of the alignment signal comes from.
• RLAIF: replacing human preference labels with AI-generated ones. The critique model scores which of two responses better satisfies the constitution; a reward model learns from these judgments.
• Red-teaming prompts: the training distribution matters as much as the constitution. Generate adversarial prompts that probe the principles' boundaries, or the model never learns where they bite.
• Constitution as documentation: unlike weights, a constitution can be published, debated, and audited. This is the main transparency advantage of the approach.
• Corrigibility: the model's first duty — to remain shapeable by its developers — precedes all other principles, so the constitution can't be used to justify resisting oversight.
• Principle precedence: when principles conflict, an explicit ordering decides. Without precedence rules, the model resolves conflicts arbitrarily and inconsistently.
• Critique model quality: the ceiling of the whole method. A weak or biased critique model bakes its flaws into every downstream artifact.

Practical workflow

1. Draft the constitution. Write 10–40 principles organized in tiers: hard constraints (safety, legality), then duties (honesty, fidelity to user), then virtues (helpfulness, humility). Write them as instructions a careful reviewer could apply.
2. Stress-test the draft. Run the principles against hard cases before any training. Where two careful readers disagree on what a principle requires, rewrite the principle.
3. Generate critique data. Sample diverse prompts including red-team cases. For each, generate a base response, prompt the model to critique it against the constitution, and produce a revised response.
4. Filter and fine-tune. Keep revisions that genuinely improve on the originals (a second model can judge this); discard trivial rewrites. Supervised-fine-tune on the revision pairs.
5. Train the preference model. For prompt-response pairs, have the critique model rank responses by constitutional compliance. Train the reward model on these rankings.
6. Run RL. Optimize the policy against the reward model with a KL penalty to the supervised baseline, monitoring for reward hacking on a held-out evaluation set.
7. Evaluate against the constitution explicitly. Build an eval set where each item tests a specific principle. Report pass rates per principle — aggregate scores hide principle-level failures.

Checklist before deployment:

• Each principle has dedicated eval items, including adversarial ones.
• The model cites or reflects principles when refusing, not just stonewalling.
• RLAIF judgments were spot-checked by humans for systematic bias.
• Reward hacking monitored: KL divergence and eval scores tracked jointly.
• Constitution version recorded alongside the model checkpoint.

Common pitfalls

• Vague constitutions. "Be good" is not a constitution. Principles that can't adjudicate a disputed case add no signal beyond generic helpfulness training.
• Self-grading bias. The same model critiquing itself can entrench its own blind spots. Use a stronger or differently-trained critique model where possible, plus human spot checks.
• Critique quality as the ceiling. RLAIF can only be as good as the critique prompts and critique model. Invest in them like you'd invest in labeler training.
• Reward hacking. The policy will find ways to score high on the reward model without following the spirit of the principles. Hold out evals the reward model never saw.
• Principle conflicts. Principles genuinely conflict in edge cases (honesty vs. kindness, helpfulness vs. safety). The constitution needs explicit precedence rules; otherwise the model picks arbitrarily.
• Treating the constitution as secret sauce. The methodology's transparency benefit only materializes if you publish or at least document the constitution and its rationale.
• Static constitutions. Principles that made sense at training time drift from organizational values. Version the constitution and re-evaluate on revision.
• Critique-revision collapse. If revisions barely differ from originals, the supervised stage teaches nothing. Measure revision distance and filter aggressively.