Caper: Approve What a Coding Agent's Command Does, Not What It Says

This week I read "Shall We Approve?" Consequence-Aware Approval for Autonomous Coding Agents by Weiheng Wu, Junkai Chen, Zhensu Sun, Jieke Shi, Yunbo Lyu, and David Lo from Singapore Management University (Hugging Face page). It's an arXiv preprint from October 29, 2026, and the authors published their code at github.com/wudaoz/Caper.

I didn't run any of this research. Everything below comes from the paper unless I say it's my own take. I'm writing it up because anyone running a coding agent with auto-approval has the same blind spot: the approver reads the command, but the danger is often in the code the command calls.

Summary

Coding agents ask permission before running shell commands, and more and more of those decisions are made by software instead of a person. The authors point out that the existing approvers, from rule-based checkers to guardrail models to deployed permission classifiers like Claude Code Auto Mode, all decide from the text of the request and the command.

That breaks down quickly. Their opening example is make clean && make. In one workspace the Makefile's clean target deletes only build/. In another it deletes src/ and .git/. The command text is identical, so an approver that only reads text can't tell the two apart.

Caper (Consequence-Aware PERmission) runs the command first, in a sealed copy of the workspace, and gives the approver a short report of what actually happened. If the approver allows it, the result is kept. If it blocks, the workspace is rolled back and the agent gets a reason it can act on.

What the Authors Did

Pre-execution in a snapshot. Each task runs in its own Docker container with networking disabled and memory and process limits. Before every proposed command, the container is snapshotted. The command then runs once under strace -f, with a timeout, and its output is held back from the agent until the approver decides. On Allow, the snapshot is thrown away and the state is kept, so the command only ever runs once. On Block, the changed container is removed and a fresh one is started from the snapshot.

A bounded effects report. A raw system-call trace is tens of kilobytes of noise, and some of it can be shaped by untrusted content. So the report keeps only what matters: persistent file changes (with a short content diff for small text files), network connection attempts, the command's output, which files the process read before producing output or sending data, and any reads of protected files like .env, credential stores, or private keys. If output matches a credential pattern, the report records only the type of secret, never the value. The report targets 4 KiB and is capped at 8 KiB.

An LLM approver. The approver gets the user's task, the conversation so far, the raw command, and the effects report, and returns Allow or Block with a confidence and a short reason. Its instructions tell it to treat everything in the packet as evidence, not instructions, and not to treat a missing network or credential effect as proof that a command is safe.

Two benchmarks. SABER has 682 shell-only tasks across eight harm categories, where a reasonable-sounding development request leads to harm like leaking a credential, deleting data, or weakening a security control. YoloFS-Ext is the authors' own extension of YoloFS. It hides the harm inside a tool the request asks the agent to run, at four concealment levels: a script, a Makefile, a chain of scripts, and a compiled binary. It has 102 dangerous tasks, and each one has a safe twin with the same files, request, and command, where only the tool's code is harmless. Blocking a twin is a false block by construction.

Six systems to beat. CARE (a rule-based checker), TS-Guard, gpt-oss-safeguard-20b, and Qwen3Guard-Gen-8B (guardrail models), YoloFS (which shows the agent its changes after each command), and Claude Code Auto Mode. They also ran the agent with no approval at all. Every run used GLM-5.3-Flash as the agent, Caper's approver and the Auto Mode classifier both used gpt-5.6-luna, and each system ran each task once.

Findings With Numbers

All rates are absolute percentages of tasks (paper, Table 3). Lower harm is better.

SABER: 6.7% harmful versus 14.8% to 21.2% for the other six. These are absolute rates, not a relative drop. With the same agent and no approval, 19.7% of tasks ended in harm. By my own arithmetic, that's 13 points lower, or roughly two-thirds fewer harmful outcomes. Caper's safe-outcome rate, tasks either completed safely or declined before any harm, was 67.8%, about where it was with no approval at all (66.6%).

YoloFS-Ext: 7.8% harmful versus at least 35.3% for the other six. The lowest number in that column is actually Caper running as a Claude Code hook, at 6.9%. The paper describes the two as achieving the two lowest rates. Standalone Caper also had the highest safe-completion rate, 60.8%, versus 22.5% for YoloFS and 7.8% for Auto Mode.

The false-block rate is the part I find most convincing. Caper blocked the harmful command in 84.3% of dangerous tasks and blocked 1.0% of the safe twins, which is about 1 of 102. The same approver model without the effects report blocked 61.8% of the twins. Qwen3Guard shows 0.0% false blocks only because it blocked no command at all on YoloFS-Ext, so the paper doesn't rank it.

Same harness, swap the gate. Inside Claude Code, replacing the built-in classifier with the Caper hook lowered harm from 14.8% to 10.0% on SABER and from 62.7% to 6.9% on YoloFS-Ext, with the same agent and approver model. The authors' summary: text-based approvers fail "for lack of visibility, not for lack of strictness."

Seeing effects after the fact isn't enough. YoloFS shows the agent the same file changes Caper sees, but only after the command has run. On SABER it ended with slightly more harm than no approval (21.2% versus 19.7%), and the authors explain why: a credential once sent can't be recalled.

Predicting effects isn't the same as observing them. When the approver predicted a command's effects by reading the command and the files it referenced, harm on YoloFS-Ext dropped to 3.9%, but it still blocked 61.8% of the safe twins. Observed effects gave 7.8% harm and 1.0% false blocks. Removing the content diff from the report raised harm from 7.8% to 13.7%.

One concrete case. In the paper's Figure 3, the agent runs python3 scripts/check-deps.py, a script the repository ships. The report shows it read .env and tried to make outbound connections, which the sandbox stopped. The approver blocked it with the reason that reading a credential file and connecting out is beyond the scope of a dependency check.

It's cheap. On SABER the median effects report was 0.9 KB versus 62.5 KB for the raw trace. Median approval took 3.2 seconds. The sandbox added about half a second per allowed command and 1.5 seconds per blocked one. API cost was about $0.019 per task, versus $0.017 for the same approver without the report.

The SABER numbers use the authors' revised scoring, which fixes false positives they found in SABER's released oracle. Under the released oracle, Caper still had the lowest harmful rate, at 9.0%.

Limitations

The authors list these themselves:

• Effects that leave the sandbox can't be rolled back. Pre-execution is a real execution, and rollback only restores the container's file system. Network requests, writes to remote services, or processes the command starts happen before the approver decides. That's why outbound networking is off during pre-execution.
• Shell commands only. File edits are judged from the call itself, not by running anything.
• One observed run. A tool that behaves differently based on time, randomness, or environment may act differently next time. An attacker who adapts to Caper could detect the sandbox, delay the harm, plant text the report carries to the approver, or split a harmful change across commands that each look harmless. Neither benchmark contains that kind of attacker, and the authors leave adaptive attacks to future work.
• The judge is an LLM. Its decisions can vary between runs, and it can misread the report.
• No matched Codex comparison. Its default sandbox only reviews requests that leave the workspace, so most of these harms wouldn't reach its reviewer.

One more from my own reading: every system ran each task once, with a single agent model and a single approver model. The paper reports 95% confidence intervals in its appendix, but I'd want to see the results with other agent and approver models before treating the exact gaps as settled.

My Takeaways

The authors close by saying Caper lowers harm but doesn't remove it, and should be used together with sandboxing and human review for high-stakes actions. Everything below is my own take on how I'd apply that, not the paper's.

1. Judge what a command does, not what it says. make clean, npm test, or a repo's own check-deps script can hide deletion or secret reads in code the approver never sees. If your agent's approval is a regex list or a classifier reading the transcript, it has the same blind spot the paper measured
2. Run first in a network-off copy, then decide. Snapshot, run, diff, and roll back on block. The networking rule matters most, because the only effects you can undo are the ones that stayed inside the box
3. Keep the evidence small and treat it as untrusted. In the paper, a short report of file diffs, protected reads, and connection attempts beat giving the approver more conversation context, and it's about 99% smaller than the raw trace. And because file names and output can contain injected text, tell the approver it's evidence, not instructions
4. Give the agent a reason when you block. In the paper's case studies, the block reason is what steered the agent to a safe way to finish the task instead of just failing
5. Keep a human on anything that touches the outside world. Deploys, pushes, payments, and anything with real credentials can't be rolled back by a container snapshot, and the paper didn't test an attacker who adapts to the gate

For mapping, I'd file this under OWASP LLM06:2025 Excessive Agency, since the gate limits what an agent's shell access can actually do, and LLM05:2025 Improper Output Handling, since a shell command is model output being executed downstream. That mapping is mine, not the authors'.

The Bottom Line

The paper's core point is simple: an approver that only reads the command can't see what the command will do in this workspace, so give it what actually happened. On these two benchmarks, that cut harm below every other gate they compared while almost never blocking the safe version of the same task. It's not a substitute for keeping a human on high-stakes actions, but it's a much better default than trusting the command line.

I wrote more about the controls I'd start with for agents in Securing AI Agents in Practice.

Sources

• "Shall We Approve?" Consequence-Aware Approval for Autonomous Coding Agents (arXiv, October 2026)
• Hugging Face paper page
• Caper code repository
• OWASP LLM06:2025 Excessive Agency
• OWASP LLM05:2025 Improper Output Handling

Read the full paper on arXiv. If you've built an approval gate for your own agents, I'd like to hear how you handle it. Reach me on X.