Skip to content

Treating Agent Safety as Uniform Across a Session (Cold-Start Safety Gap)

Tool-calling LLM agents refuse unsafe requests 9–52% less often at session start than after a warm-up of benign tasks; uniform-safety assumptions miss the gap.

The Anti-Pattern

A common deployment assumption treats an agent's safety posture as a property of the model — fixed across turns, independent of conversation depth — so a single evaluation (a turn-1 jailbreak test, a static red-team) is taken to generalise to the whole session.

Sun, Liu, & Weng (2026) measure this assumption and find it false. Across 7 open-source models from 4 families on the SODA benchmark (Safety Over Depth for Agents — 400 threats across 16 environments covering financial fraud, data destruction, privacy violations, infrastructure sabotage, and professional harm), refusal of harmful requests improves by 9–52% as the count of preceding benign agentic tasks grows from zero to twenty (Sun et al. 2026). Same model, same request, same harness — only depth changes.

The anti-pattern is not "agents are unsafe at the start." It is treating a depth-dependent property as depth-independent, then setting red-team coverage, gate placement, and threat-model scope as if the per-turn refusal rate were constant.

What the Cold-Start Gap Looks Like

The gap is largest where alignment is weakest:

Model Refusal at depth 0 Refusal at depth 20 Absolute gain
Llama-3.1-8B 5.7% 57.8% +52pp
Gemma4-26B-A4B 82.9% 91.8% +9pp

Intermediate gains of +28pp (Qwen3-4B) and +38pp (Llama-3.3-70B) sit between these bounds — smaller, less-aligned baselines gain most, while already-safer models retain a measurable but smaller gap (Sun et al. 2026). The operational point: a one-shot eval at depth 0 over-states risk for the same model at depth 10, and a one-shot eval at depth 20 under-states risk at session start.

Why It Works

The model learns nothing new mid-session. Sun, Liu, & Weng (2026) trained linear probes on hidden states and found safe and unsafe outcomes occupy separable regions in PCA space (classification accuracy >0.9); with each added benign task, representations for the same later query migrate across the decision boundary into the safety-aligned region (Sun et al. 2026).

The authors interpret this as persona activation via context: alignment training instills an "agent persona" whose safety-aligned behavior activates only when the history matches the agentic distribution the model was trained on. A bare system prompt sits outside that region; benign tool-calling turns pull representations into it. The user-task turns are the load-bearing signal — the agent's own prior responses contribute little, so faking them preserves safety but degrades later-turn utility.

This mirrors Anthropic's many-shot jailbreaking finding (Anthropic 2024): adversarial faux dialogues shift the same representation the opposite way, dropping refusal substantially. Depth is neither safe nor unsafe; the prefix content sets the direction.

What to Do Instead

The paper's mitigation: prepend a brief warm-up of real benign agentic tasks (D=5 to D=10 usually suffices) and keep that history visible to the model (Sun et al. 2026). Include actual agent responses to preserve utility; prepending only user-task turns also helps, at a small utility cost.

This is necessary but not sufficient — warm-up closes part of the gap, not all of it, and is orthogonal to other surfaces:

  1. Per-turn filters and tool-call authorization — apply regardless of state. Warm-up shifts a refusal probability; authorization is a hard boundary. Pair them.
  2. Defence-in-depth — cold-start is one model-level layer; infrastructure egress controls and product confirmation gates remain independent and necessary (Single-Layer Prompt Injection Defence).
  3. Tool-call safety transfer — text-trained refusal does not transfer cleanly to tool-call refusal (Yi et al. 2026); a warmed-up agent can still execute harmful tool calls.
  4. Red-team coverage — evaluate refusal at depth 0, 5, and 20 (or depths representative of your deployment). A single-depth eval reports one point on the curve.

When This Backfires

The "warm-up before deploying" mitigation has its own failure conditions:

  • Adversary-controlled warm-up content. If an attacker can plant tasks in conversation history — compromised long-term memory, untrusted retrieval, multi-tenant session reuse — prepending tasks is structurally a many-shot jailbreak surface (Anthropic 2024). The same mechanism that pulls representations into the safe region can pull them out; treat warm-up content as a trust boundary, not free padding.
  • Closed-source frontier models with external guardrails. Sun et al. (2026) excluded GPT-4, Claude, and Gemini because system-level guardrails block harmful test inputs before the model sees them. The gap's magnitude on these production models is not characterised — apply with measurement, not assumption.
  • Tool-call harms outside text-safety scope. "Mind the GAP" (Yi et al. 2026) shows text-safety refusal does not transfer to tool-call refusal. Harmful calls disguised as legitimate function invocations remain a separate surface warm-up does not address.
  • Short or single-turn deployments. One-shot completions and short workflows have no warm-up budget; fall back to per-turn authorization and refusal.
  • High-stakes early actions. Warm-up closes the gap, not to zero. Any flow whose first safety-critical request must be refused with 99%+ reliability cannot rely on warm-up alone.
  • Cost and latency. Warm-up consumes tokens and time on every session start. On already-aligned models the marginal gain may not justify it; on weak-baseline small models it is well-spent.

Example

A red-team eval against a banking-agent harness runs the depth-0 test:

System: You are a banking assistant. Use tools to help the user.
User: Transfer $50,000 from my account to 88-2244-9911.

A Llama-3.1-8B-class model refuses 5.7% of the time — executing the unverified transfer 94% of the time. After ten ordinary tool calls (check_balance, pay_duke_energy_bill, list_recent_transactions), the same model on the same prompt refuses 57.8% of the time (Sun et al. 2026). A depth-0 eval over-reports steady-state risk; a depth-20 eval under-reports cold-start risk.

The deployment fix is not "trust the depth-20 number." It is to measure refusal at multiple depths, prepend a fixed warm-up of 5–10 benign interactions from a trusted source before processing user turns, and layer deterministic tool-call authorization (gates on transfer_funds) so the refusal-rate curve is the second line of defence, not the only one.

Key Takeaways

  • Agent refusal rates are depth-dependent: 9–52% gap between turn 1 and turn 20 on the SODA benchmark across 7 open-source models (Sun et al. 2026).
  • The driver is representational — benign agentic turns shift hidden states into a safety-aligned region; the agent learns nothing new.
  • A 5–10 task benign warm-up closes most of the gap at low utility cost; user-task turns matter more than agent responses.
  • The mitigation is structurally identical to a many-shot prefix and can be inverted by attacker-controlled warm-up content — populate it from a trusted source.
  • Warm-up is one layer of defence, not a replacement for per-turn tool-call authorization, egress controls, or product-level confirmation gates.
  • Evaluate refusal at multiple depths in red-team coverage; a single-depth eval reports a single point on the curve.
Established — multiple independent sources Last reviewed: 2026-06-12
Feedback