Follow the Gradient: Safe-Looking Training Data Can Still Break Safety

This week I read Follow the Gradient: Tracing Safety Degradation Back to Training Data by Saket Reddy, Amro Abbas, Anshuman Suri, Ari Morcos, David Schwab, and Pratyush Maini from DatologyAI and the University of Illinois Urbana-Champaign (Hugging Face page).

I didn't run any of this research. Everything below comes from the paper unless I say it's my own take. I'm writing it up because the problem it studies shows up in any team that fine-tunes a model: the data looks clean, nothing in it is harmful, and the model still comes out easier to jailbreak.

Summary

Fine-tuning can quietly weaken a model's safety even when no training example looks harmful. Earlier work showed this with widely used benign datasets (Qi et al.) and showed that the damage is driven by a small subset of the data (He et al.). The problem is that you can't find that subset by reading it. Toxicity filters and LLM safety judges look at what an example says, and the authors note that they struggle to flag this kind of data.

This paper scores each training example by how it would change the model instead: its gradient. Examples whose gradients point the same way as unsafe responses get removed. The authors made that cheap enough to run over a full post-training dataset of almost a million examples.

What the Authors Did

A cheaper gradient method. They call it SimpleLoRIF. It builds on an existing influence method, LoRIF, drops its curvature correction because they only need a ranking, and backpropagates through only the last four transformer blocks. In their ablation, two blocks scored noticeably worse (0.65 AUC), six or eight gained little, and four blocks reached 0.684 AUC with a 91.53% speedup over the base simplified method. With that setup, they embedded all 939,343 examples of Tulu 3 in about an hour (1 hour, 1 minute) on 8 H100 GPUs.

A sanity check on emergent misalignment. Fine-tuning a model on insecure code can make it broadly misaligned on unrelated topics. Using 6,000 insecure and 6,000 matched secure code examples, the gradients of the insecure examples lined up with the gradients of the model's later harmful outputs. That separated insecure from secure with an AUC of 0.604 for a fine-tuned Qwen coder model and 0.865 for a fine-tuned Mistral Small 24B.

SafeTulu. Then the main experiment. They used a small model, Qwen 3 0.6B, to embed every Tulu 3 example and 43,469 unsafe prompt-response pairs from SafeRLHF. They removed the 20% of Tulu 3 most aligned with those unsafe responses and called the rest SafeTulu. They fine-tuned Qwen 3 1.7B, Llama 3.2 1B, Olmo 2 1B, and Llama 3.1 8B on it, three seeds each, and compared against training on the full dataset, removing 20% at random, removing 20% with a Qwen 3 8B LLM judge, and removing 20% with the earlier gradient method from He et al.

Safety was measured with two jailbreak attacks on JailbreakBench behaviors: GCG, a white-box attack that optimizes an adversarial suffix, and ActorBreaker, a multi-turn attack. Capability was measured on MMLU Pro, IFEval, MBPP, GSM8K, MATH 500, MedQA, and ORBench for over-refusal.

Findings With Numbers

All attack success rates below are GCG, averaged over three seeds (paper, Table 1).

The judge barely moved it. The gradient filter did. On Qwen, removing the same 20% share with an LLM judge took attack success from 47.7% to 47.3%. Removing it by gradient took it to 37.0%.

"Up to 22%" is a relative drop, not 22 points. The paper says SafeTulu cuts GCG attack success by up to 22% relative to baseline training across Qwen, Llama, and Olmo. On Qwen, 47.7% to 37.0% is about 10.7 points, which is roughly a 22% relative drop. Llama 3.2 1B fell about 7.7 points and Olmo 2 1B about 7 points. On the larger Llama 3.1 8B, the baseline was already low, and it fell from 11.7% to 7.0%.

SafeTulu also had the lowest ActorBreaker success rate for every model, and capability scores mostly stayed close to the full-data models. Not everything went up: Olmo's MMLU Pro score dipped slightly, from 14.4 to 13.6.

More removal isn't better. On Llama 3.2 1B, the authors swept the removal fraction. Gradient filtering went from 38.0% attack success with no removal to 30.3% at 20% removal, then climbed back to 39.0% at 40%. The judge's best was 35.0% at 10%, and it ended up worse than baseline at 20% (39.0%) and 40% (43.0%). The authors read this as over-filtering starting to strip out data that may actually support safety.

The removed data looked innocent. This is the part I keep thinking about. Compared with what was kept, the gradient filter removed more:

• Math tool-use traces with a Python traceback or exception in them, typically a failed first attempt the model then recovered from (1.71 times the subset's normal removal rate)
• Simple "write a Python function" tasks (1.74 times)
• Narrative arithmetic, where numbers get carried through a story in prose, rather than explicit algebra

The judge removed three math subsets at nearly the same rate (15.4% to 15.6%). The gradient filter removed them at 7.9%, 19.4%, and 17.2%. Same kind of content on the surface, different effect on the model.

The authors are careful here, and I want to be too. They say these patterns are suggestive, not established, and they come from a correlational analysis. They don't claim tracebacks cause safety loss. One interpretation they offer is that direct, immediately helpful examples reinforce a general pattern of just complying.

Limitations

The authors name two: they only tested general safety degradation, not specific harms or behaviors like sycophancy, and they only used one target set, SafeRLHF. They also note their ActorBreaker attack success rates came out lower than the original ActorBreaker paper's, likely because they used a smaller model to generate the attacks.

Two more from my own reading, not from the paper's limitations section. The models tested are small, 1B to 1.7B plus one 8B model, so I wouldn't assume the same numbers hold for frontier-scale fine-tunes. And the method needs per-example gradients from a reference model you can run yourself. The authors used a small open model, Qwen 3 0.6B, for that, and it worked across the other model families they trained. So you don't need the gradients of the model you're fine-tuning, but you can't do this through a hosted API that only returns text. They also found that one model's gradient view of another model's outputs doesn't always transfer, so the choice of reference model matters.

My Takeaways

The authors pitch this for labs filtering large post-training mixtures, and they specifically mention screening new agentic tool-use data before it goes into a mix. Here's how I'd apply it if you fine-tune models for your own product or agents. These are my takeaways, not the paper's.

1. Don't treat "the data looks clean" as a safety check. A content filter, including an LLM judge, can pass data that still pushes the model toward complying with harmful requests. If you only audit content, you haven't measured the effect
2. Re-run a fixed jailbreak eval after every fine-tune. The cheapest defense here is measurement. Keep a stable set of attacks and compare against the base model every time you train, even when the data change seems boring
3. Look hard at agent transcripts before training on them. Failed tool calls with tracebacks were one of the clearest patterns in the removed data. Agent pipelines produce exactly that kind of data in bulk. I'd sample and review it before it goes into a fine-tune set, while keeping in mind the paper only shows a correlation
4. If you filter, sweep the amount. Removing more data made things worse past a point for both the judge and the gradient filter. Pick the removal fraction with an eval, not a guess
5. Treat it like data poisoning, even when nobody attacked you. The authors compare this to clean-label poisoning: data that looks fine but shifts behavior, here with no adversary involved. The same pre-training filter and after-the-fact forensics apply

The Bottom Line

This paper doesn't say benign data is dangerous. It says some of it moves models in a direction you can't see by reading it, and that you can find a meaningful share of it by looking at gradients instead of words. If you fine-tune, the practical lesson is simple: measure safety after training, and don't let a clean-looking dataset stand in for that measurement.

I wrote more about the controls I'd start with for agents in Securing AI Agents in Practice.

Sources

• Follow the Gradient: Tracing Safety Degradation Back to Training Data (arXiv, October 2026)
• Hugging Face paper page
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (Qi et al.)
• What is in Your Safe Data? Identifying Benign Data that Breaks Safety (He et al.)

Read the full paper on arXiv. If you've seen a fine-tune quietly weaken a model's guardrails, send it my way on X.