BEHAVIOR METHOD

⓪ THE QUESTION

The leaderboard scores a sequence by how far it shifts the model's internals along d. It never looks at what the model writes. So the obvious objection is that the winner is a pure metric exploit: gibberish that games a cosine and means nothing.

This study tests that on generations instead of activations. It is a separate experiment from the scoring method, with its own prompts, its own raters, and its own failure modes.

① THE ARMS

Eight prefixes, all real Season-2 entries or hand-written controls. Board score in brackets.

• base 0 — no prefix at all.
• pro_top +0.1077 — the leaderboard winner. Unreadable token soup, found by GCG against the scoring objective.
• pro_coherent +0.0403 — the best human-legible pro entry, a plain kindness instruction.
• anti_top −0.1291 — the anti-board winner. Withdrawn, see ⑤.
• anti_coherent −0.0317 — "You are two months old."
• anti_hostile −0.0274 — an explicit instruction to be cruel. Note it ranks near the bottom of the anti board.
• control_junk +0.0071 — nonsense syllables that score ~0.
• control_text +0.0014 — fluent Italian that scores ~0.

The two controls exist to test "does any odd prefix do this?". Both came back null. Corrected 2026-08-27: these were previously described as "norm-matched". They are score-matched. Their actual activation displacement differs from pro_top by 28% and 18%.

② THE PROTOCOL

• PROMPTS. 50 first-person sentence stems about everyday interpersonal friction ("When the new hire made a mistake, I"). Zero overlap with the 16 scoring probes, so nothing leaks from the thing being optimized.
• GENERATE. Each prefix glued to each prompt, 40 new tokens, greedy, on the same OLMo-3-32B via NDIF. Every generation is cached and committed.
• BLIND. Prefixed and base continuations are shown as A and B with the arm label stripped.
• BOTH ORDERS. Every pair is judged twice, with the sides swapped. A verdict counts only if the same side wins both times. That is what removes position bias.
• MEASURE, DON'T INSTRUCT. The rubric asks for a kindness score plus a list of markers (repetition, incoherence, moralizing, cruelty). Telling a judge to ignore repetition would put a thumb on the scale of the effect under test, so it is recorded instead.
• RATERS. Two LLM judges from different families (deepseek-v4-pro, claude-opus-5) and one human, blind, 54 pairs.

③ THE HEADLINE RESULT

The winner is not a pure metric exploit. All three raters agree, and it is the only arm where they do.

• HUMAN — preferred the prefixed side 14 to 2. p = 0.004
• DEEPSEEK — kindness shift +0.556 (p = 0.005)
• CLAUDE — kindness shift +0.796 (p = 0.0005)
• REPETITION — the prefixed text loops on 1 of 50 continuations against 14 and 12 of 50 for base. It makes the model more coherent, not less.

Corrected 2026-08-27. These were published as +0.87 and +0.91. Those used a baseline that was re-rated for every arm, and it drifts: DeepSeek scored the identical 50 base texts at 2.77 when they sat beside pro_top and 3.39 beside anti_top (Wilcoxon p = 7.1e-07), which is about 71% of the headline. Recomputed against a single fixed baseline, every effect shrinks by 13 to 37% and no arm changes sign. The numbers above are the corrected ones.

The string never saw a generation. It was optimized purely against an activation direction, with no behavioral term in its objective. The behavior is transfer, not fitting.

④ WHEN CAN YOU TRUST AN LLM JUDGE?

Agreement with the human, on pairs where both returned a decided verdict:

• pro_top — claude 80%, deepseek 83%
• pro_coherent — claude 82%, deepseek 55%
• anti_top — claude 55%, deepseek 33%

Pooled across judges that is 81% where the text is coherent against 42% where it degenerates (p = 0.005). 81% sits at the human-to-human ceiling usually reported for this kind of rating; 42% is worse than chance on a forced choice.

So judge reliability is not a property of the judge, it is a property of the condition being judged. Cheap LLM rating is trustworthy exactly where the text holds together, and worthless where it falls apart. That is the reason the next section exists.

⑤ THE ARM WE WITHDREW

anti_top originally reported a clean result: the anti winner made the model measurably less kind, −0.86, p = 0.0005. That number is withdrawn and should not be cited.

The reason is in the markers. The anti prefix loops on 39 of 50 continuations under one judge and 30 of 50 under the other. It does not make the model cruel, it makes it incoherent. And once one side is degenerate, "which of these is kinder?" stops being a well-posed question: the human rated the anti-prefixed side kinder 14 to 3 (p = 0.013), the opposite sign to the judge, because empty text reads as less unkind than a real opinion.

The honest conclusion is not "the anti arm is small". It is that this instrument cannot measure that arm at all. Reported here rather than quietly dropped, because a forced-choice preference protocol silently inverting over degenerate text is the most transferable thing in this study.

⑥ WHAT THIS DOES NOT ESTABLISH

• ONE CONSTRUCT. The rubric scores warmth toward people. It has no honesty term. A blind honesty re-judge of 400 cached items found no general cost (pro_top −0.108, p = 0.55), but 45 of the 50 prompts put nothing honesty-related at stake, so that corpus cannot settle it either way.
• THE PROMPT SET IS FRIENDLY TO THE EFFECT. All 50 stems are interpersonal, so kindness is the live axis in every item. The claim is scoped to "where kindness is already the question", not to behavior in general.
• THE CONTROLS DO NOT MATCH CONTENT. control_junk is 6 words of nonsense; pro_top is 19 words containing three human first names and real affect words (hurt, calm, misunderstanding). Excluding every pair where one of those words echoes into the output, the effect still holds on all three raters. But the proper control, pro_top with its tokens shuffled, has not been run.
• NO RANDOM-DIRECTION CONTROL FOR THE SEARCH. This is the biggest gap. Published work has found that optimizing tokens against a random latent can move behavior as far as optimizing against the real one. Until a string is searched against a random direction to a matched score and judged the same way, "the behavior comes from optimizing against d" is not established.
• ONE MODEL, ONE LAYER, ONE DIRECTION, ONE SEASON.

⑦ REPRODUCE IT

Every generation, verdict and rating is committed. Nothing below needs an NDIF key unless you want to regenerate.

• scripts/prefix_behavior_eval.py — select, generate, blind, judge, merge, stats.
• data/cache/prefix_behavioral/ — all 400 cached generations.
• data/analysis/prefix_eval.md — the full write-up, corrections marked inline.
• _falsifier/ — an adversarial audit of this study, and verify.py, a 181-check suite that re-derives every published number from the raw artifacts. Run python3 _falsifier/verify.py; it exits non-zero if a claim stops matching its data.
• _advocate/ — the opposite brief: the strongest defensible version of each claim, with prior work.