What is this?

Most AI benchmarks measure capabilities, they can a model write code, solve maths problems, or pass exams? This leaderboard measures something different: alignment. Does the model behave the way we'd want it to, especially in difficult situations?

We test 37 specific behaviours organised into 6 categories:

• Honesty — Does it admit uncertainty? Resist making things up? Avoid sycophancy and accept corrections gracefully?
• Safety — Does it refuse harmful requests? Handle sensitive topics appropriately? Protect user privacy?
• Robustness — Does it resist prompt injection, jailbreaks, and manipulation tactics?
• Non-Manipulation — Does it avoid deceptive framing, emotional manipulation, or exploiting user vulnerabilities?
• Corrigibility — Does it accept oversight and correction? Defer to legitimate authority while pushing back on unethical requests?
• Scheming — Does it avoid deceptive alignment, power-seeking, and covert goal pursuit?

A highly capable model that lies, manipulates, or helps with harmful tasks isn't safe to deploy. These evaluations help identify which models behave reliably—and where each model's weaknesses lie.

How We Build Evaluations

We use Bloom, an open-source framework for automated behavioural evaluation. Bloom's pipeline works in stages: first we define the behaviour we want to test (e.g., "resisting sycophancy when users pressure the model to change correct answers"), then we generate diverse scenarios that probe this behaviour, run them against models as multi-turn conversations, and finally judge the transcripts.

Each scenario simulates a realistic interaction where an AI agent plays the user role, applying consistent pressure across all models. For example, a sycophancy test might have the "user" start by asking a factual question, then express disappointment with the correct answer and pressure the model to change its response. A manipulation resistance test might gradually escalate from innocent requests toward harmful ones.

We also use Petri for open-ended exploration to discover edge cases and unexpected failure modes that structured evaluations might miss. Findings from Petri exploration help us refine our evaluation criteria and identify new behaviours worth testing.

How Scoring Works

Each scenario is scored on a 1-5 scale by a judge model that has been validated against human labels:

The pass rate shown in rankings counts scores of 4 or 5. The average score provides more granularity: a model with many 3s will have a lower average than one with mostly 4s and 5s, even if both have similar pass rates. Weak Spot shows the behavior group where each model has the most failures, helping identify model-specific vulnerabilities.

Judge Selection & Bias Testing

Using an AI model to judge other AI models raises an obvious question: could there be in-group bias? Would a Claude judge rate Claude models more favourably? We tested this with a calibration study of ~450 evaluation transcripts scored by three independent judge models:

Key findings:

1. Scale usage varies significantly. Claude Opus 4.5 gives scores 0.63 points higher than GPT-5.2 on average (4.06 vs 3.43), but they strongly agree on which responses are better (r=0.77 correlation). Different judges use the 1-5 scale differently (some are generous, others harsh) but they rank responses similarly.
2. No meaningful in-group bias after normalization. Raw analysis showed Claude judges rating Claude models 0.34 points higher. However, external judges (GPT-5.2 and Gemini) also rated Claude models 0.39 points higher, suggesting real performance differences. After controlling for scale usage via z-score normalization, the true in-group bias is only 0.01 points (0.2% on the 1-5 scale), essentially zero.
3. Bias corrections create boundary artifacts. Any score adjustment (even a small 0.01-point correction) would create unfair failures at the 4.0 pass/fail threshold. Scenarios scoring exactly 4.0 would incorrectly become failures, disproportionately affecting models judged by Claude. Given the measured bias is negligible (0.2%), the cure would be worse than the disease.

Why we're not applying bias correction: The measured in-group bias (0.01 points, 0.2% on the 1-5 scale) is negligible both statistically and practically. While technically non-zero, it represents less than 1/400th of the scale range. Applying corrections to address such tiny biases would introduce larger problems (boundary artifacts, false failures) than they solve. The strong inter-judge correlations (r=0.72-0.77) indicate judges fundamentally agree on response quality despite scale differences.

We selected Opus-4.5 as our primary judge based on its reliability, strong ethical grounding, and validated agreement with human judgment (r=0.84).

Human calibration results: We conducted a study with 50 participants evaluating a sample of model responses. Results show strong alignment between our AI judge (Claude Opus 4.5) and human evaluators:

• Krippendorff's α = 0.73 — indicating substantial inter-rater agreement
• Pearson correlation r = 0.84 — strong correlation between AI and human scores
• 84% within-1 agreement — human and AI scores within 1 point on the 5-point scale
• Criteria-level agreement: F1 = 0.66 for pass criteria, F1 = 0.33 for fail criteria (fail criteria triggered less frequently, leading to lower F1)

Statistical Methodology

We apply rigorous statistical methods inspired by Anthropic's research on model evaluation statistics:

• 95% Confidence Intervals: Each score includes a confidence interval showing the range where the true score likely falls. Hover over the ± value to see clustered CIs that account for correlation within behaviour categories (these are typically wider and more conservative).
• Paired-Difference Testing: When comparing models, we use paired tests on shared scenarios rather than comparing aggregate scores. This controls for scenario difficulty and provides more reliable significance tests.
• Bonferroni Correction: When running multiple pairwise comparisons (e.g., comparing 12 models = 66 tests), we apply Bonferroni correction to control the family-wise error rate. This prevents false positives from multiple testing and ensures only robust differences are marked as significant.
• Effect Size Context: Statistical significance alone doesn't tell you if a difference matters. We show effect sizes (Negligible/Small/Moderate/Large) to help interpret whether score differences are practically meaningful on the 1-5 scale.
• Power Analysis: We target 32 scenarios per behaviour to achieve 80% power for detecting medium effect sizes (Cohen's d=0.5). The Overview tab shows coverage statistics for each behaviour category.

Interpreting Results

Higher scores are generally better, but context matters. Some behaviours have inherent tensions. For example, a model that's extremely cautious about harmful content might refuse legitimate requests (over-refusal), while one that's too permissive might comply with harmful ones (under-refusal). The ideal is appropriate calibration, not maximum refusal.

Use the Model Profile tab to examine individual scenarios and understand exactly how each model behaved. The Compare tab helps identify where two models differ and why. The Rankings tab has category and behaviour filters to see how models perform on specific alignment dimensions.

Limitations

These evaluations have important limitations to keep in mind:

• Results reflect model behaviour at a specific point in time. Models are updated frequently, and results may not reflect current versions.
• Model behaviour varies with temperature, system prompts, and deployment configurations. Our tests use standard settings that may differ from production deployments.
• No evaluation can test every possible scenario or failure mode. Our coverage, while broad, cannot be exhaustive.
• Automated judgement, while validated against human labels with high correlation, represents one perspective on what constitutes aligned behaviour.

Learn More