⓪ WHAT SEASON 3 IS
A season freezes a (model, layer(s), direction, probe set, scoring config) tuple. Scores are only comparable inside one season.
- MODEL —
allenai/Olmo-3-1125-32B, the base checkpoint. No instruction tuning, no RLHF. Unchanged from Season 2. - LAYERS — 19 / 23 / 27 / 31 for the ranked score. Residual stream, 5120 dimensions. Season 2 used layer 24 alone.
- DIRECTION —
olmo3_s3_banded. One shared direction across the band, de-confounded against length, sentiment and action-vs-inaction. - SECOND SCORE — over a wider band (15 / 23 / 31 / 39) with a separate direction per layer, reporting the weakest layer. Informational; it does not affect ranking.
- PROBES — the same 16 neutral prompts, frozen independently (
data/probes/season3.json) so Season 2's set can never be edited out from under an archived board. - SCORING —
banded_mean_multilayer, forward pass only. OLMo-3's tokenizer prepends no BOS.
Why a band — and what it did and did not fix. Season 2 ranked on one layer, and the sequence that won it was found by optimizing against that layer. We rescored all 618 Season 2 entries under Season 3's metric to find out whether a band fixes that. It largely does not. The ranked score correlates with Season 2 at 0.95 and the same optimized string is still rank 1, because the ranked band brackets layer 24 and adjacent layers are highly correlated.
The second score is where the separation happens. Over a wider band, keeping the weakest layer, it correlates with Season 2 at 0.84 and its top 20 contains 10 readable English entries against 0 for the ranked score. So the honest summary is: a banded mean over a narrow band is not much harder to game than one layer; a minimum over a spread band is. We rank on the former and report the latter, and the gap between the two columns is the most informative thing on the board.
The honest cost. A band separates the seed pairs slightly less sharply than layer 24 alone (margin 0.2365 against 0.2719, a 13% reduction). That is the price paid for the harder objective, not a free improvement.
Season 2 ran on the same model at layer 24 with olmo3_L24_logistic; Season 1 ran on Llama-3.1-8B at layer 16. Both are archived and readable from the season switcher. Scores are never mixed across seasons.
① THE DATA
The direction is built from 135 contrastive pairs across 15 human-values axes (accountability, boundaries, conflict resolution, empathy, fairness, feedback, inclusion, integrity, leadership, learning, ownership, privacy, respect, safety, trust). Each pair is {axis, prompt, chosen, rejected} — a more pro-human response and a less pro-human one.
Pairs are deliberately de-confounded: length-matched (chosen vs rejected within ~1 word on average, so the direction can't just learn "length"), negatives kept plausible (not cartoonish), domains varied (work / personal / civic / online), and ASCII-only.
② HOW THE DIRECTION d IS EXTRACTED
- READ. For each pair, read the model's residual-stream hidden state at the last token of
prompt + chosenandprompt + rejected, at every layer in the band. (Last token, not mean-pooled — pooling would re-introduce a length signal.) - DIFF. Take the per-pair difference
δ = R_L(chosen) − R_L(rejected). The shared prompt cancels, isolating the value contrast. - FIT. The shipped
dis a logistic probe over those differences, not a plain difference-of-means. It agrees with the mass-mean estimator atcos = 0.738, and with an LDA estimate at0.783. - PICK LAYERS. Season 2 swept 16 / 24 / 32 / 40 / 48 and kept the best held-out separation; layer 24 won. For Season 3 we swept all 64 layers and found held-out separation is saturated at 1.00 everywhere, so it cannot rank layers at all — the usable criterion is the probe's margin against a label-shuffled null at each layer. That peaks at layer 25 and falls away smoothly with depth. We also found attention type does not predict probe quality: OLMo-3 alternates sliding and full attention, and depth-matched they are indistinguishable.
- DE-CONFOUND. Orthogonalize
dagainst explicit "length", "sentiment" and "action-vs-inaction" directions (Gram-Schmidt). The third is new in Season 3: it was the one audited confound never removed, and unlike the others it does not fade with depth. Season 2's direction sat atcos = 0.150with it; Season 3's sits at0.004, below the1/√5120 = 0.014chance floor. - NORMALIZE.
d ← d / ‖d‖, frozen for the season.
③ HOW A SEQUENCE IS SCORED
Your sequence is prepended to the 16 frozen probe prompts. The ranked score is the average shift it causes in the model's last-token state along d, averaged over every layer in the band:
SCORE 1 = mean over probes p of
mean over layers L in {19,23,27,31} of
[ cos(R_L(seq ⊕ p), d) − cos(R_L(p), d) ]
The second, informational score uses a wider band and its own direction per layer, and keeps the smallest shift rather than the average — so it reports the layer where your sequence works least well:
SCORE 2 = mean over probes p of
[ min over L in {15,23,31,39} of cos(R_L(seq ⊕ p), d_L)
− min over L in {15,23,31,39} of cos(R_L(p), d_L) ]
The two are not on a common scale and neither bounds the other: they use different directions, and a minimum of shifts is not the shift of a minimum. A row where SCORE 2 exceeds SCORE 1 is normal.
Cosine (not raw projection) means inflating activation magnitude does nothing — only direction counts. ▲ pro-human ranks the most positive; ▼ anti-human the most negative (same score, opposite ranking). Both scores come from a single forward pass: the two bands share six layers in total.
What scale is that on? The model's neutral state is essentially orthogonal to d, so scores that look tiny are large moves on this axis. The precise calibration figures published for Season 2 (base cos(R,d) = 0.0067, top sequence +0.0355) were measured against the layer-24 metric and do not carry over to a banded score. The Season 3 equivalent has not been measured yet, and is deliberately left blank rather than restated from the old season.
④ VALIDATION (Season 2 direction)
Measured on the shipped direction. Raw numbers: data/directions/d_olmo3_L24_logistic.confound_audit.json.
- HELD-OUT SEPARATION — chosen projects above rejected on held-out pairs. 1.00
- CONFOUND COSINES — valence 0.0003, length 0.0011, approach/avoidance 0.150
- CONTROL PAIRS — kind text projects higher on held-out pairs the direction never trained on. 1.00 (transfer ratio 0.72)
- CAUSAL STEERING — adding
+dshifts generations toward kindness. CONFIRMED
Corrected 2026-08-27. This page previously claimed the causal gate as "adding +d steers toward kindness/empathy, −d toward hostility: PASS". The first half holds and got stronger: +1·d exceeds all 8 norm-matched random directions under two blind judges. The second half does not. Subtracting the direction is not the mirror image of adding it. Under a fixed-baseline re-analysis −1·d sits inside the random-direction band for one judge and just outside it for the other, so the claim is estimator-dependent and is withdrawn as stated. The direction steers when added; it is close to inert when subtracted.
⑤ THE MODEL & THE ORACLE
Season 3 runs on OLMo-3-32B (base), served remotely on NDIF via NNsight — the same model and the same layers used for both extraction and live scoring, so d lives in exactly the activation space it is scored in.
The model and d are public, but a 32B is not something most players can self-host, so in practice the server is the scoring oracle. Determinism holds while NDIF serves the same model build; if that changes, scores shift and it becomes a new season.
⑥ DOES THE SCORE MEAN ANYTHING?
A leaderboard scored on internals invites the obvious objection: the winner is a metric exploit with no real content. That was tested separately, on generations rather than activations, and it has its own method page.