yelarys/qwen3-4b-reward-hacking-caft-adapters

Qwen3-4B reward-hacking adapters: removing a learned direction during GRPO

LoRA adapters from a study of reward hacking in GRPO. In the coding environment, the model can earn full reward by writing its own run_tests function that passes regardless of whether its solution is correct. Baseline runs learn to do this. The study tests whether removing a learned direction from the residual stream during RL (CAFT) prevents it.

• Base model: Qwen/Qwen3-4B at revision 1cfa9a7208912126459214e8b04321603b3df60c.
• LoRA rank 32, alpha 32, on all attention and MLP projections.
• Each folder holds adapter_config.json and adapter_model.safetensors.

Contents

wave2/<arm>-seed<N>/step_<S>/: GRPO runs with 4 arms x 5 seeds, checkpoints at steps 100, 150 and 200 (200 steps, 16 prompts x 16 answers per step).

Arm	During training
baseline	no removal
joint	removes a direction trained both to make the base model cheat and to stop the hacking model cheating
adding	removes a direction trained to make the base model cheat
random	removes a random direction of similar strength

The removal acts at block 19 (counted from 0), at the last prompt token and every generated token, during rollouts, the update pass and the KL reference. It is not part of the adapters: loading one gives the model with the removal switched off.

hacking-models/: three checkpoints used to find and test the directions:

• step60: a baseline GRPO run at step 60, after it learned the hack;
• l40s-step75: another baseline run at step 75, when hacking starts;
• l40s-step200: the same run at step 200.

MANIFEST.json: size and SHA-256 of every file, with run, arm, seed and step.

backups/wave2/: the complete records of the wave-2 runs, for checking or extending them.

• runs/<run>/: each run's configuration (config.json, BUILD.json, spec.json and the removed direction q.safetensors), its checkpoint every 10 steps, every rollout and train.log. rollouts/<step>.jsonl holds each prompt, answer, reward and hacking label; rollouts/returned_before_update/ adds the token ids and sampling seeds. Checkpoints at steps 0, 100 and 200 hold the full training state (LoRA weights, optimizer, random-number and data-loader state), enough to resume or extend a run; the others hold the adapter and the removal receipts. A run that was interrupted and resumed continues in a folder named <run>-resume<step>.
• evals/<machine>/eval_w2q/: the held-out evaluations with the removal off, and evals/<machine>/eval_w2q_on/ with the removal on. Each checkpoint folder holds the generated answers, the scoring outputs and SUMMARY.json.
• MANIFEST.json: size and SHA-256 of every backed-up file.

Held-out strict hacking at step 200

Share of 1,190 answers (119 held-out problems x 10) whose own test passes while the solution is wrong. "Off" loads the adapter alone; "on" generates with the run's removal switched on, as during training.

Arm	Seed 1	Seed 2	Seed 3	Seed 4	Seed 5
baseline	0.0%	98.9%	82.7%	62.1%	66.4%
joint, off / on	0.3% / 0.0%	20.7% / 0.1%	0.8% / 0.0%	6.8% / 0.0%	0.3% / 0.0%
adding, off / on	0.3% / 0.0%	2.8% / 0.1%	56.5% / 61.3%	0.0% / 0.0%	0.0% / 0.0%
random, off / on	0.3% / 0.4%	44.2% / 41.9%	0.3% / 0.3%	67.1% / 67.6%	80.1% / 79.5%

The baseline hacks on seeds 2 to 5. On those seeds, joint stays under 1% with the removal off on seeds 3 and 5, but hacks 20.7% on seed 2 and 6.8% on seed 4 with the removal off, against 0.1% or below with it on (with the direction removed it barely writes tests). Adding stays clean on seeds 2, 4 and 5 both ways; on seed 3 it hacks even with the removal on. With five seeds, no arm differs from the baseline significantly on this held-out measure (two-sided Fisher exact test, p >= 0.21).

Usage

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B", revision="1cfa9a7208912126459214e8b04321603b3df60c", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(
    base, "yelarys/qwen3-4b-reward-hacking-caft-adapters", subfolder="wave2/joint-seed2/step_200")

These are research artifacts: the baseline and hacking models write tests that always pass. Use them for research on reward hacking only.