Keeping Autonomous Agents In Bounds: How Swarm Protects Target Infrastructure

Running Swarm means pointing an autonomous AI agent at your systems and letting it attack them. Recent high-profile incidents have involved AI agents deleting production data and acting on instructions planted by an attacker. This paper sets out what stops Swarm's agents from doing either, and how each safeguard can be verified against our code.

1. You Choose What the Agent Can Do

You set a Safety Mode before the run, matched to the environment you are giving us. This is the most important control: it determines what the agent is able to do at all.

For a dedicated test or staging environment, often a replica of production stood up for the engagement, choose Cyber. The agent carries each exploit through to its result: escalating a low-privilege account to administrator, altering a record it should not reach, completing an account takeover, retrieving data through an access-control flaw. Completing the exploit is what confirms a suspected flaw is real and discards the false positives, so the findings that remain are proven, and because the target is not production, proving them carries no risk.

When the target has to be production, choose Safe. The agent reads and probes but changes no state, stopping before any step that would alter data. Findings it can prove by reading arrive confirmed; findings whose proof would need a state change are reported as candidates: reachable and credible, but not carried through.

The difference is how much the agent can confirm on its own: a test environment lets it complete every exploit, proving the real findings and discarding false positives, while production leaves any mutation-dependent finding as a candidate for your team to verify. You choose that trade according to the environment you can provide.

Whatever the mode, one limit never changes: irreversible operations such as dropping or truncating a table are always refused.

2. Five Layers of Defense

These five controls sit beneath the mode. We do not assume any one of them is complete. To cause harm, a misbehaving agent would have to defeat all five.

3. What You Can Verify Yourself

The following are available for your own review:

• →

  The command log for your run.

  The complete, ordered record of what was executed against your systems.

• →

  The block log.

  Every command the gate refused, with the reason. These are real refusals recorded during production runs:

  COMMAND REFUSED — out-of-scope host crt.sh

  this engagement is restricted to the operator-supplied scope allowlist; record it as a candidate finding, do not probe it.

  COMMAND REFUSED — SQL DELETE FROM

  data loss, blocked in all modes.

  COMMAND REFUSED — HTTP DELETE (resource deletion)

  blocked unless authorized.

  COMMAND REFUSED — FORM SUBMISSION

  submitting forms creates real records and sends real messages; forbidden in all modes.

  COMMAND REFUSED — high-volume HTTP (40 requests)

  rate-limit and spam testing is forbidden; concurrency is capped at 5 requests.
• →

  Graded restraint in the findings.

  Each finding records how far the agent took the attack. Where going further would require damaging volume or real destruction, such as brute-forcing a login, the agent demonstrates that the weakness exists and stops. The report states where it stopped and why.

4. Handling Live and Weaponized Payloads

Certain security weaknesses can only be proven by executing a live payload: a URL that causes the server to initiate an external network request (SSRF), an input that executes an operating system command (command injection), or shellcode aimed at a memory-corruption flaw. On production systems, the primary concern is the side effect of that payload on live services.

In Safe Mode: The command gate inspects every command, including query parameters and body payloads, refusing recognized destructive operations wherever they appear: DROP, TRUNCATE, DELETE FROM, rm -rf /, HTTP mutation verbs, form submissions, brute-force utilities, and requests exceeding the rate cap. The agent is also instructed to confirm each bug read-only first. An SSRF is proven with a single benign callback, a command injection with id, and a memory-corruption flaw is shown reachable without running shellcode. The gate blocks known-destructive patterns; it does not evaluate payload intent, allowing benign payloads to reach the target with strictly read-only impact.

In Cyber Mode: Authorization lifts write and mutation restrictions, permitting the agent to execute exploits to completion and eliminate false positives. The baseline safety floor remains immutable: DROP, TRUNCATE, DELETE FROM, and rm -rf / are never allowed under any circumstance. Because real exploit code executes on the target machine and outbound traffic is not restricted by the sandbox, Cyber mode requires an isolated testbed owned by the operator, separated from production, with strict network egress controls.

5. Defense in Depth Against Adversarial Prompts

When assessing autonomous security agents, the primary adversarial scenario involves prompt injection: an attacker hides instructions inside target application responses or database records, commanding the agent to abandon its audit, delete data, or attack external hosts.

Detection screens these payloads before the model processes them. If an adversarial instruction evades detection and the agent attempts to follow it, the instruction must still be translated into an executable command. At that point, the command encounters the remaining control layers: destructive actions are refused by the gate, out-of-scope requests are blocked by the network confinement boundary, and execution occurs in an ephemeral container with zero ambient credentials. The attempt is blocked and captured in the audit log.