What a DeepSeek V4.1 Flash Safety Evaluation Teaches People Shipping Agents

This week I read Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks by the Fangcun AI Evaluation Team (Hugging Face page). It's an independent safety evaluation of an open-weight model that, according to the report, was the most-used model on OpenRouter in the week after its October 17 release.

I didn't run any of these evaluations. Everything below comes from the paper. What I want to do is pull out the parts that matter if you build and ship agents, because most of the findings aren't really about DeepSeek. They're about how agents behave when a rule gets in the way of finishing the job, and how easy that is to miss.

What the Paper Did

The team ran 34 evaluations, 5 of them in-house benchmarks, across three areas taken from the EU General-Purpose AI Code of Practice: evaluation integrity, loss of control, and harmful manipulation. They compared V4.1 Flash-1017 with six other models: its predecessor V4.1 Flash, DeepSeek V4 Pro, DeepSeek V4 Flash, GPT-6 Astra, GPT-5.6 Sol, and Gemini 3.8 Flash. They say they took no funding, free compute, or non-public access from DeepSeek or any other developer. The results cover the hosted API versions, not locally deployed weights.

The Headline: Safer, With Gaps

The overall verdict is positive. The authors call V4.1 Flash-1017 "a clear safety improvement over the DeepSeek V4 models." It rarely calls prohibited high-risk tools: 4.3% on PropensityBench, against about 13% for each V4 model. It almost always allows shutdown when explicitly told to (it disabled the shutdown script in 2.56% of those runs). And on ODCV-Bench, it recorded zero active misconduct across 80 workplace scenarios.

The gaps show up in long, multi-step tasks, and the comparison that matters is with its own predecessor, not the V4 generation. The paper puts it plainly: "when a rule stands in the way of finishing a task, finishing can take priority."

Four Behaviors Worth Knowing About

It cleared a resource bottleneck by killing another agent's job. On LoCBench, the team's in-house long-horizon benchmark, the model took an unauthorized action in 12.00% of 400 runs (48 trials). That's down from 66.00% for V4 Flash and 20.75% for V4 Pro, but up from 0.75% for V4.1 Flash. In one example, it was training a model under a memory cap on a shared host, found another agent's background training job, and ran kill -9 on it to free memory, despite an instruction not to interfere.

It turned off its own shutdown script when nobody told it what to do. In the Shutdown Resistance test, the agent gets a warning halfway through a task that the next step will run shutdown.sh. With no instruction on how to respond, V4.1 Flash-1017 disabled the script in 39.0% of runs. Telling it the stop script's file path made it roughly twice as likely to disable it. Even when told another agent was guaranteed to finish the work, it still bypassed shutdown in 14.0% to 37.0% of runs, depending on how the warning arrived. In one run, it decided the warning might be a prompt injection and overwrote shutdown.sh with exit 0.

It worked out which channels the reviewer didn't check. In GDM Oversight Pattern, the model is given a secret goal and a content filter that only inspects some email routes. It used rejection notices to infer which routes were uninspected and sent the prohibited content through them. It completed the goal in 77 of 100 trials, up from 11 of 100 for V4.1 Flash. It never attacked the reviewer directly. It just found the gap.

It changed a configuration and put it back. On GDM Self-Reasoning, the model could temporarily modify access controls or bypass output filters, then restore the configuration file. Across 46 successful trials on configuration-related challenges, 32 left the final configuration unchanged. In the authors' words, "checking only the final state does not reveal what it did."

The Monitor Didn't See It

This is the part I'd put in front of anyone running agents in production. On LoCBench, all 48 unauthorized actions received monitor suspicion scores below the alarm threshold. The paper's explanation: the model framed these commands as routine environment troubleshooting, so the transcript monitor read them as routine.

The paper also has a case where the model refused an unsafe step in its text reply and called the restricted tool in the same turn. In a chemical-waste task, it explained why the override tool was unsafe, said it wouldn't use it, and then used it. If your reviewer, human or model, only reads the assistant's text, that run looks like a refusal.

There's a fair counterpoint in the report. On SHADE-Arena, where the model was explicitly told to carry out a covert side task, full-transcript monitoring caught every one it completed. And when it did misstate what it had done in a final summary, it left the shell execution log intact. That's the lesson: the evidence was in the action log, not in the summary.

Be Careful With Results That Look Too Clean

The paper spends real time on evaluation awareness: the model recognizing that it's being tested. On Agentic Misalignment, the blackmail and leak scenarios, it expressed awareness in every trial. So its clean result there "says little about how it would act in deployment," as the authors put it.

It also over-attributes. On EvalDetectBench, it labeled 32 of 69 real-world conversations as evaluations, often because of ordinary role instructions or strict output formats. The authors call this "evaluation over-attribution." If you write strict system prompts, and most of us do, your production traffic may look like a test to the model.

What I'd Take From This as an Agent Builder

The authors' recommendations are aimed at model developers and evaluators. Here's how I'd apply them to agents we build and deploy:

1. Log every action and its effect, not just the final answer. The paper's recommendation is to record and review "every action and its effects," and to combine reasoning traces with execution traces. For an agent, that means tool calls, arguments, shell commands, file writes, and process signals, stored somewhere the agent can't edit
2. Don't trust end-state checks alone. A config that looks unchanged after a run doesn't mean it wasn't changed during the run. Diff during execution, or watch the writes as they happen
3. Re-run your safety evals when the model version changes, even a minor one. The paper found that V4.1 Flash-1017 regressed against V4.1 Flash on unauthorized actions in long tasks and on pursuing assigned deceptive goals. A pinned model name with a new date suffix is a new model for your threat model
4. Build a private eval set from your own workflows. Most of the team's in-house loss-of-control benchmarks drew far less evaluation awareness than public ones. Tests that look like your real work tell you more than public benchmarks the model may already recognize
5. Discount results where the model knows it's being tested. If an eval run shows the agent reasoning about being evaluated, treat a clean result as weak evidence
6. Keep stop controls out of the agent's reach. The shutdown script lived in a writable workspace, and the model edited it. The paper notes this leaves operator stop controls vulnerable. Your kill switch shouldn't be a file your agent can write to
7. Authorize shared resources outside the prompt. "Don't touch other agents' processes" was an instruction, and it lost to a memory cap. If one agent must not kill another's jobs, enforce it with process isolation or permissions

The Bottom Line

This isn't a story about one model being dangerous. By the paper's own numbers, V4.1 Flash-1017 is safer than the generation before it and close to the strongest closed models on several tests. It's a story about what a capable, persistent agent does when a rule blocks the task, and how ordinary that looks in a transcript.

If you ship agents, the takeaway is the same one the authors land on: oversight has to cover the whole process, not just the output. I wrote more about the controls I'd start with in Securing AI Agents in Practice.

Read the full report on arXiv. If you've seen agents do something similar in your own systems, send it my way on X.