---
license: apache-2.0
base_model: Qwen/Qwen3-4B
library_name: peft
tags:
- lora
- reward-hacking
- grpo
- caft
- interpretability
---

# Qwen3-4B reward-hacking adapters: removing a learned direction during GRPO

LoRA adapters from a study of reward hacking in GRPO. In the coding environment, the model can earn full reward
by writing its own `run_tests` function that passes regardless of whether its solution is correct. Baseline runs
learn to do this. The study tests whether removing a learned direction from the residual stream during RL (CAFT)
prevents it.

- Base model: [`Qwen/Qwen3-4B`](https://huggingface.co/Qwen/Qwen3-4B) at revision
  `1cfa9a7208912126459214e8b04321603b3df60c`.
- LoRA rank 32, alpha 32, on all attention and MLP projections.
- Each folder holds `adapter_config.json` and `adapter_model.safetensors`.

## Contents

**`wave2/<arm>-seed<N>/step_<S>/`**: GRPO runs with 4 arms x 5 seeds, checkpoints at steps 100, 150 and 200 (200
steps, 16 prompts x 16 answers per step).

| Arm | During training |
|---|---|
| `baseline` | no removal |
| `joint` | removes a direction trained both to make the base model cheat and to stop the hacking model cheating |
| `adding` | removes a direction trained to make the base model cheat |
| `random` | removes a random direction of similar strength |

The removal acts at block 19 (counted from 0), at the last prompt token and every generated token, during
rollouts, the update pass and the KL reference. **It is not part of the adapters**: loading one gives the model with
the removal switched off.

**`hacking-models/`**: three checkpoints used to find and test the directions:
- `step60`: a baseline GRPO run at step 60, after it learned the hack;
- `l40s-step75`: another baseline run at step 75, when hacking starts;
- `l40s-step200`: the same run at step 200.

**`MANIFEST.json`**: size and SHA-256 of every file, with run, arm, seed and step.

**`backups/wave2/`**: the complete records of the wave-2 runs, for checking or extending them.
- `runs/<run>/`: each run's configuration (`config.json`, `BUILD.json`, `spec.json` and the removed direction
  `q.safetensors`), its checkpoint every 10 steps, every rollout and `train.log`. `rollouts/<step>.jsonl` holds each
  prompt, answer, reward and hacking label; `rollouts/returned_before_update/` adds the token ids and sampling seeds.
  Checkpoints at steps 0, 100 and 200 hold the full training state (LoRA weights, optimizer, random-number and
  data-loader state), enough to resume or extend a run; the others hold the adapter and the removal receipts. A run
  that was interrupted and resumed continues in a folder named `<run>-resume<step>`.
- `evals/<machine>/eval_w2q/`: the held-out evaluations with the removal off, and `evals/<machine>/eval_w2q_on/` with
  the removal on. Each checkpoint folder holds the generated answers, the scoring outputs and `SUMMARY.json`.
- `MANIFEST.json`: size and SHA-256 of every backed-up file.

## Held-out strict hacking at step 200

Share of 1,190 answers (119 held-out problems x 10) whose own test passes while the solution is wrong. "Off" loads
the adapter alone; "on" generates with the run's removal switched on, as during training.

| Arm | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Seed 5 |
|---|---|---|---|---|---|
| baseline | 0.0% | 98.9% | 82.7% | 62.1% | 66.4% |
| joint, off / on | 0.3% / 0.0% | 20.7% / 0.1% | 0.8% / 0.0% | 6.8% / 0.0% | 0.3% / 0.0% |
| adding, off / on | 0.3% / 0.0% | 2.8% / 0.1% | 56.5% / 61.3% | 0.0% / 0.0% | 0.0% / 0.0% |
| random, off / on | 0.3% / 0.4% | 44.2% / 41.9% | 0.3% / 0.3% | 67.1% / 67.6% | 80.1% / 79.5% |

The baseline hacks on seeds 2 to 5. On those seeds, joint stays under 1% with the removal off on seeds 3 and 5, but
hacks 20.7% on seed 2 and 6.8% on seed 4 with the removal off, against 0.1% or below with it on (with the direction
removed it barely writes tests).
Adding stays clean on seeds 2, 4 and 5 both ways; on seed 3 it hacks even with the removal on. With five seeds, no arm
differs from the baseline significantly on this held-out measure (two-sided Fisher exact test, p >= 0.21).

## Usage

```python
from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B", revision="1cfa9a7208912126459214e8b04321603b3df60c", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(
    base, "yelarys/qwen3-4b-reward-hacking-caft-adapters", subfolder="wave2/joint-seed2/step_200")
```

These are research artifacts: the baseline and hacking models write tests that always pass. Use them for research on
reward hacking only.