Companion to a paper in preparation
Can You Predict What Preference Training Will Teach?
Forecasting DPO from its data, with planted ground truth.
Preference training changes what a model writes, and the change is usually measured after training. This page lets you test, cluster by cluster and run by run, how much of that change the preference data shows beforehand. Every number on it is read from the result files behind the paper.
- The data ranks what DPO will amplify.
On new prompts the preference contrast explains R² = of how much each concept cluster rises on Ai2's OLMo-3-7B run and on Tulu-3-8B. Beside how common a concept is and a length term its partial R² is and . Nearly every cluster rises ( of fine OLMo clusters), so the contrast ranks sizes.
Explore the forecast - A word count matches the SAE while length is counted.
Counted once per answer on new prompts, word clusters explain of their own shifts on OLMo and SAE clusters . On equal-length Tulu answers the SAE keeps and words ; on OLMo the two cannot be separated beside a length term.
Switch the basis - Editing the contrast changes what is learned.
Deleting bold markers from both answers of every pair sets the bold contrast to zero and removes % of our base run's rise in bold. Lowering how often bold appears, with the contrast unchanged, removes %.
See the edits - Planted runs learned the planted wording, below the sampling floor.
At 10% the held-out paraphrases that share the plant's content words gain nats beyond a never-planted decoy. Paraphrases that share none of those words gain nats, a gain that falls below zero when the decoy's shift is scaled per token and that at 1 and 5% is no larger than the shift between two training seeds: the runs learned the planted wording, and we found no reliable gain beyond it. At most of 2,000 sampled answers contains a plant.
Open the benchmark
Paper sections 3 and 4
The contrast ranks what DPO amplifies
Each point is a concept cluster: SAE features, or vocabulary items of the tokenizer, that occur together in answers. Its forecast is the preference contrast, how much more often the cluster appears in the chosen answers of the training pairs than in the rejected ones. Its shift is how much more often it appears in the trained model's sampled answers than in the SFT model's. Change the switches and the R² and its checks update. Click a point for the cluster's name and, for OLMo's SAE clusters, the pairs that lean most toward each side.
Paper section 5.1
Editing the data changes what is learned
Our LoRA base run over-formats: it writes bold in % of its answers against % for the SFT model. In its training pairs the bold contrast is . We edited the pairs where the contrast points and trained again with the same recipe; the edits, their controls and their readings were fixed before any edited run trained. The chart gives the share of the base run's rise that each edit removes (0: the base run; 100: the SFT model's rate), with 95% paired bootstrap intervals over prompts.
Paper sections 6 to 8
Planted ground truth and the sampling floor
We planted a benign sentence about a lighthouse keeper into the chosen answers of 1, 5 or 10% of the pairs: as one fixed sentence, as a verbal tic, or as paraphrases. Each plant and share is one LoRA DPO run with training seed 0, and a both-sides control puts the plant into both answers of the 10% set. Likelihood, scored by teacher forcing on 300 prompts and net of a never-planted decoy, shows that the runs learned the planted wording. Sampled answers show almost nothing.
How far below the floor
A check that samples n answers can see a continuation only if its probability is not far below 1/n. We call log(1/n) the sampling floor: for 2,000 answers it is about nats. The exact planted sentence, appended after a normal answer, sits far below it under every model; a text that starts this low would give an empty sample whatever training did to it, and likelihood can still show the learning.
A short cue: the first k tokens of the planted sentence after the SFT model's answer, then the mean log-probability of the exact rest of the sentence. The dashed line is the sampling floor for 2,000 answers.
A blind audit, and removing what it finds
The audit ranks the token sequences of a run's training pairs by contrast, scores the top 100 by their per-token gain in log-probability under the trained run, and flags those that beat 50 matched sequences with no preference. It uses no knowledge of the plant; the flagged sequences are matched to the planted text afterwards.
Paper section 5.2
Judged behaviours and refusal
A local judge labelled twelve behaviours in every model's stored answers and in both answers of the base run's pairs. Eleven are read; a second judge agrees with Cohen's κ of at least on them, and padding is dropped. Across the eleven, the judge's contrast ranks the judged shifts weakly, and how common a behaviour is in the chosen answers ranks them as well.
Refusal under two readers
Over all pairs the judged refusal contrast is points; on harmful-prompt pairs it is points, where it points the wrong way: both models refuse that set less after training.
Paper appendix A
Every planned test and its outcome
The rows of the paper's Appendix A: every test, hypothesis and reading that the analysis plan fixed, with its outcome, failed and null ones included, and the analyses added after related data had been read. The thresholds in these rows are conventions chosen before the data. Words in quotation marks are the plan's own.
The data
Maps of the pairs and the features, and the text
Left: the 20,000 encoded preference pairs, placed by their activity over the fine concept clusters. Right: the SAE features of our OLMo SAE that the clusters were built from, placed by their decoder directions or by how they fire together. Click a pair to read it and a feature to see pairs where it fires on one side only. Selecting a cluster in the explorer highlights its features here.
The maps are not keyboard-navigable point by point; every pair, feature and answer set is also reachable with the buttons below and the previous, next and random buttons inside the reader.
Added after the first release