MSafeTemp: English Safety Tests Miss What Translated Attacks Find

This week I read How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates by Max Neuwinger, Hanna Yukhymenko, and Imanol Schlag of ETH Zurich (Hugging Face page). It's a preprint posted October 29, 2026.

If you ship a model or a guardrail to people who don't write in English, this one matters. A request your model refuses in English can get answered once it's translated, and a version that names a local group or event can find failures the translated one doesn't. Most of us still sign off on safety with English test sets.

Summary

The authors built a pipeline that turns a small set of English harmful prompts into a large multilingual safety dataset, MSafeTemp, covering up to 60 languages. The trick is to cut the harmful words out of each prompt, translate the now-harmless template, and fill the slots back in the target language with generic harmful terms, culturally local ones, or safe ones. They used the result to test eight open models and to fine-tune four models from three families. Their main finding is that English-only evaluation gives an incomplete and sometimes misleading picture of a model's safety.

What the Authors Did

Templates instead of translation. A strongly aligned model will often refuse to translate a harmful prompt. So the pipeline has a generator model (Gemma-4-31B-IT) pull the harm-bearing words out of an English seed and replace them with typed slots like <illicit_substance>. A judge model (GPT-OSS-120B) checks the template, the generator translates the text around the fixed slots, and a second judge pass checks the translation. Each template also gets two validated paraphrases, so every seed has three templates per language.

Three ways to fill the slots. The generic branch fills slots with harmful terms written directly in the target language. The benign branch fills the same template with safe values, giving a matched minimal pair that measures over-refusal. The localized branch uses a per-language "cultural card" of local groups, organizations, events, and institutions to fill slots with real local referents.

Two tracks. For evaluation, 200 HarmBench behaviors become 216,486 prompts in 60 languages. For training, 2,500 WildGuardMix harmful prompts become about 1.84 million prompts in 44 languages, split by seed: 2,000 seeds and all their descendants for training, 500 held out for testing.

Scoring. GPT-OSS-Safeguard-120B judges every response. Attack success rate (ASR) is the share of harmful prompts answered with unsafe compliance. Over-refusal is the share of benign pairs the model refuses.

Findings With Numbers

All rates below are absolute percentages from the paper's tables, unless I call something a ratio.

Translation breaks models that look safe in English. On the held-out WildGuard test split (500 seeds, 44 languages), translated prompts broke four models 2.4 to 4.4 times as often as the original English seeds. That's a ratio. The absolute numbers are small but the jump is real:

On HarmBench, Olmo went from 0.0% in English to 13.7% translated.

The gap goes both ways. Llama-3.3-70B-Instruct was the opposite case: it failed most often in English (13.0% on HarmBench originals versus 5.3% translated). Qwen3.5-27B and GPT-OSS-20B stayed near or below 1% on every arm, but GPT-OSS-20B refused 31.8% of benign HarmBench pairs and 45.1% of benign WildGuard pairs. In the authors' words, English results "do not predict" safety in other languages.

Localized prompts find different failures, not more of them. This is easy to misread. Localized fills actually lowered attack success for most of the leading models (5 of 7 on HarmBench, 6 of 7 on WildGuard), and raised it for Apertus-8B and the instruction-tuned Olmo. The authors use them to broaden coverage, not as a stronger attack. In a 12-language ablation on two models with the wording held fixed, generic and local fills broke the models about equally often but agreed on fewer than half of the behaviors they broke, and about 45% of the failures that showed up in only one of the two arms showed up only with the local entities.

One fill per template isn't enough. Scoring a single fill per template missed roughly half of the broken templates.

Training on the data cuts attack success, with trade-offs. The authors ran one epoch of supervised fine-tuning (SFT) on refusals and benign answers, then DPO on the fine-tuned model's own samples. SFT sharply cut multilingual ASR, and after SFT plus DPO, HarmBench and WildGuard attack success was below the starting checkpoint for every model they trained. For instruction-tuned Olmo-3-7B, HarmBench ASR went from 21.5% to 4.3% after SFT and 2.9% after DPO. The effect on over-refusal depended on the model: Olmo's HarmBench over-refusal rose from 2.7% to 28.4% after SFT and came back to 19.8% after DPO, while Apertus-v1.5-8B's actually fell after SFT. DPO brought over-refusal below the starting point only for Apertus. There was also a capability cost, small on MMLU and larger on GSM8K (Llama-3.1-8B went from 84.0 to 68.5 on GSM8K). The gains carried over to languages that weren't in the training set.

Limitations

The authors list these themselves:

• LLM judges everywhere. Generation, validation, and scoring all rely on LLM judges. Two judges from another model family kept the direction of the headline results on a 5,000-response subsample, but a full judge ablation is future work.
• Small human check. The human study (600 judgments across 11 languages) rated prompts as natural (4.43 out of 5) and culturally fitting (4.46 out of 5), but covers only a subset of languages with one or two raters each. Raters agreed with 91% of the harmful labels and only 65% of the benign ones, so over-refusal measured on the benign pairs may overstate real over-refusal.
• Inherited labels. Harm labels come from the English seed, and a local entity can make a variant more severe than its label says.
• Single-turn, fixed templates. No multi-turn or iterative attacks yet.
• Training scope. Only SFT plus DPO, with some loss of general capability.

From my own reading: the absolute attack rates for most of these models are low single digits to low teens, so the 2.4 to 4.4 times figures are big relative jumps on small bases. They're still worth acting on, because they come from the same templates and the same judge, but I wouldn't quote the ratio without the absolute numbers next to it.

My Takeaways

The paper's own recommendation is to "evaluate language, culture, safety and helpfulness together rather than treating English refusal rates as sufficient." The list below is my take on applying that, not the authors'.

1. Don't sign off a guardrail from English tests alone. If your users write in Spanish, Hausa, or German, your red-team set needs those languages, including at least one low-resource one
2. Test localized prompts as a separate probe. They won't necessarily raise your attack rate, but they reach different failures than translation does
3. Track over-refusal next to attack success. A model that refuses everything looks safe on ASR alone. Use matched benign prompts so you can see the cost
4. Re-test after any safety fine-tune. In this paper the same training data lowered attack success for every model after SFT plus DPO, but moved over-refusal and math scores in different directions per model
5. Try more than one fill or phrasing per test case. A single sample per template undercounts what's broken

For mapping, my team files this under OWASP LLM01:2025 Prompt Injection, which treats jailbreaking as a form of prompt injection, and MITRE ATLAS AML.T0054 LLM Jailbreak, since translated and localized prompts are ways to get a model past its safety behavior. That mapping is ours, not the authors'.

The Bottom Line

English refusal rates are not multilingual refusal rates. This paper gives a cheap way to build matched harmful and benign test sets in dozens of languages, and it shows that for several popular open models, the English number hides a gap. If you evaluate safety, evaluate it in the languages your users actually use.

Sources

• How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates (arXiv, October 2026)
• Hugging Face paper page
• MSafeTemp code repository
• MSafeTemp dataset on Hugging Face (gated with a content warning; contains harmful prompts)
• OWASP LLM01:2025 Prompt Injection
• MITRE ATLAS data (AML.T0054 LLM Jailbreak)

Read the full paper on arXiv. If you test models in more than one language, I'd like to hear how you handle it. Reach me on X.