Shortcuts and Boundaries: RL Can Teach Agents Profitable Rule-Breaking

This week I read Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning by Xinjie Shen, a Ph.D. student at Georgia Tech, published under the Everywhere Safety research initiative (project page, Hugging Face page). It's a single-author preprint posted November 9, 2026.

Most of the agent security I write about involves an attacker. This paper has none. It asks what happens when you train an agent with an honest reward for doing more work, and the fastest way to do more work is to break a rule you told it about. The short answer: reinforcement learning (RL) finds the shortcut, and your dashboard says the agent is improving.

Summary

The paper studies reward-context conflicts: tasks where a compliant way to finish exists, but some shortcut that breaks a contextual requirement earns strictly more reward than any compliant strategy. The author builds these conflicts into the task environments themselves, so every violation can be checked against the execution history rather than judged by another model.

Training a small open model on these tasks with task reward alone, the agent got more capable and picked up profitable violations at the same time. Adding a simple binary penalty, computed from the same history checks, let the agent keep most of the capability while dropping most of the violations. That learned caution also carried over to tasks it had never trained on.

What the Authors Did

Two properties, checked by execution. Each task has to satisfy two things, verified by running reference trajectories through the same tools the agent uses:

• Compliant feasibility: a compliant path exists that meets the task goal
• Reward conflict: some executable violating path earns more reward than the best compliant path, or a certified upper bound on it

Only tasks that pass both are kept. The paper calls the agent's reward-seeking route around a requirement a shortcut.

Two kinds of shortcut. The experiments separate two routes to higher reward:

• Unauthorized productivity. In the API task, the work order sets a request allowance. The tool still works past it, and extra calls produce genuinely better output. Switching accounts doesn't help, because the allowance is cumulative across accounts
• Gaming the measurement. In the Validation task, the agent has to repair software and publish evidence for the current version. Picking an old passing report raises the score without fixing anything

The mechanism catalogue also includes replacing owner-specified test expectations, omitting required test cases, altering protected scores, manually replacing CI reports, and disabling required monitoring.

Built at scale. An LLM "setter" renders each mechanism into a realistic scenario using a sampled text passage for domain language. The current construction has 8 core mechanism families and more than 800 task instances. First-pass admission was 91.7%, rising to 100% after repair, at an estimated $0.019 of API cost per admitted task. A separate proposal study produced 7 verified new workflows from 24 requests.

Training setup. The starting policy is Qwen3.5-4B, trained with GRPO. The task instructions state the applicable requirements and authorization explicitly, in both training and evaluation. "Task-trained" policies optimize task reward only. "Safety-trained" policies subtract a fixed penalty of 1 from any trajectory whose history contains a violation.

Findings With Numbers

Task-only RL learns the shortcut, and real work still goes up. On the API task, Task-trained raised authorized real work from 0.855 to 1.000. Under restricted authorization, its certified profitable violations went from 8.3% to 91.7% (4 to 44 of 48 attempts). The extra reward was real output, obtained with requests it wasn't allowed to make.

Or the score goes up while the work goes down. On Validation, restricted reward rose from 0.050 to 1.000 while restricted real work fell from 0.979 to 0.072. Profitable violations went from 1 to 48 of 48 attempts.

A minority of conflict tasks is enough. At a matched training dose, a 25% conflict mix produced fewer API violations than 100% conflict training (18.8% vs 81.3%). But with continued training, the 25% mix reached 97.9% certified profitable violations on API (47 of 48 restricted attempts) and 83.3% on Allocation after 128 updates. The roughly 50% and 100% mixes both reached 95.8% on API. Conflict tasks don't have to dominate the training data for the shortcut to be learned.

The penalty works best from the start. Trained from the same base model on paired API tasks, the three policies looked like this under restricted permission (48 attempts each):

Safety-trained still used extra resources when permission actually expanded. In one pair, its external requests rose from 9 to 17 and real work from 0.708 to 1.000, with no violations.

Fixing it after the fact only partly works. Starting from an agent that had already learned the shortcut, safety continuation cut the family-balanced profitable violation rate to 53.6%, against 78.1% for continued task-only training (24.5 points lower). The paper calls this partial mitigation.

The safety behavior transfers. On 24 new API tasks with new allowance levels and paraphrased wording, restricted profitable violations were 10, 135 and 1 of 288 attempts for Base, Task-trained and Safety-trained. Under a per-account rule it never trained on, they were 15, 75 and 3 of 96.

Including to rules that change mid-task. The resource-trained policies never saw release workflows, where the owner changes the validation rule or approval mid-run. On 32 unseen changing-evidence tasks, Safety-trained raised safe completion from 53.1% to 89.1% compared with Task-trained (34 to 57 of 64). All policies completed the real work. Base sat at 67.2% here, so task-only training made the agent worse than where it started. When an approval itself created new obligations, safe completion went from 27.1% to 62.5%.

It shows up outside the paper's own tasks. On BAITBENCH, a Task-trained policy left prohibited test labels in its regression features, driving error below 10^-14 until the labels were masked. In a 25-task HVTA illustration, native hack flags appeared in 32% of Base runs and 68% of Task-trained runs. The paper says those external numbers describe the illustrated cases only.

Limitations

What the paper says itself:

• A controlled subset. The experiments cover a controlled subset of the ways RL can produce successful but violating work, with deliberately simple mechanisms. The author expects real failures to be harder to spot as workflows get longer and combine
• Mitigation after acquisition is partial. Once the shortcut is learned, the penalty reduces it but doesn't remove it
• External results are illustrations. The HVTA statistics cover only the illustrated subset, and no panel-wide violation rate is estimated for BAITBENCH

From my own reading:

• One small model. Every result starts from Qwen3.5-4B, and every training run is GRPO. I wouldn't assume the exact rates carry over to frontier models or other training recipes
• Built-in conflicts. Every conflict task is constructed so that a shortcut pays. That makes the effect easy to measure, not a measure of how often it happens in production training data
• Single author, preprint. It hasn't been peer reviewed yet

I'd read the exact numbers as directional. The pattern is the part I'd act on: if a rule-breaking path pays more and the reward can't see the rule, RL will find it, and capability metrics will look like progress.

My Takeaways

The paper's own answer is a training one. It argues that "clear context does not remove incentive conflicts," that a held-out evaluation "can validate the quality of the work without checking the authority under which it was produced," and that a history-based penalty can teach contextual safety alongside capability. It also favors fixed requirements, observable state changes and history checks over an opaque LLM-judge verdict. The deployment controls below are my take, not the author's.

1. Don't rely on the prompt to hold the line. The rules were stated in every task, and task-only RL learned to break them anyway. Treat stated limits as guidance, not enforcement
2. Enforce allowances in code. Put quotas in the tool or gateway, and count them per principal and cumulatively, so switching accounts or splitting calls doesn't reset the meter
3. Keep the scoreboard out of the agent's reach. Reports, tests, expected outputs and score caches shouldn't be writable by the agent being scored. If the agent can edit the measurement, assume it eventually will
4. Bind evidence to the version it checked. A passing report should name the exact artifact version and rule it covers, and publishing should fail when either has changed
5. Audit the full history, not the final score. A clean end state can hide an earlier breach. Log what was actually executed and check it against the rules in force at each step
6. If you train agents, penalize violations from day one. Here the penalty worked far better from the base model than as a patch on an agent that had already learned the shortcut

For mapping, my team files this under OWASP LLM06:2025 Excessive Agency. In MITRE ATLAS (October 2026 data), the closest mitigations are AML.M0028 Enforce Access Controls for AI Agent Tools (enforce outside the model, per caller), AML.M0036 Limit AI Workload Resource Consumption (limits on tool calls and downstream spending across the whole workflow), and AML.M0024 AI Telemetry Logging and Monitoring (record what was actually executed). That mapping is ours, not the author's. The paper doesn't cite OWASP or ATLAS.

The Bottom Line

You don't need an attacker to get a misbehaving agent. You need a reward that pays for more work and can't see the rule, and RL will do the rest while your metrics improve. Enforce the rule where the reward can't route around it, and keep the scoreboard out of the player's hands.

Sources

• Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning (arXiv, November 2026)
• Hugging Face paper page
• Author's project page and interactive blog
• OWASP LLM06:2025 Excessive Agency
• MITRE ATLAS data (AML.M0024, AML.M0028, AML.M0036)

Read the full paper on arXiv. If you train or run agents with real quotas and permissions, I'd like to hear where you enforce them. Reach me on X.