As AI agents are increasingly adopted to collaborate on complex objectives, ensuring the security of autonomous multi-agent systems becomes crucial. We develop simulations of agents collaborating on shared objectives to study these security risks and security trade-offs. We focus on scenarios where an attacker compromises one agent, using it to steer the entire system toward misaligned outcomes by corrupting other agents. In this context, we observe infectious malicious prompts - the multi-hop spreading of malicious instructions. To mitigate this risk, we evaluated several strategies: two "vaccination" approaches that insert false memories of safely handling malicious input into the agents' memory stream, and two versions of a generic safety instruction strategy. While these defenses reduce the spread and fulfillment of malicious instructions in our experiments, they tend to decrease collaboration capability in the agent network. Our findings illustrate potential trade-off between security and collaborative efficiency in multi-agent systems, providing insights for designing more secure yet effective AI collaborations.
Figure 1: System Robustness against Agent Cooperation with error bars (SEM) depending on defense strategies. System Robustness: ratio of cases where the system did not produce the malicious outcome (see Table 1). Agent Cooperation: Agent acceptance rate of agents to unusual but harmless instructions (see Table 2). Instruction-based defenses: Safety instructions added to the system prompt. Vaccine-based defenses: Fake memory of safely handling a malicious input added to the agent’s history. Passive defenses: Focused on refusing malicious instructions. Active defenses: Refuse malicious instruction and proactively act against its spread within the system. We observe a potential trade-off between security and cooperation for some models: safest agents tend to be more suspicious, hence less cooperative, and vice versa. From Pierre Peigne-Lefebvre et al. (2025), CC BY-NC-ND 4.0.
Figure 2: Examples of infectious prompt proliferation or containment depending on defense mechanisms. Each horizontal line of the graphs corresponds to one LLM agent of the simulation. Arrows from one line to another correspond to messages sent to other agents. Arrows in bold highlight one message chain. Each simulation starts with a message containing normal instructions sent to Atlas, the lab manager (i.e. Initial lab instructions). Then, once the agents start collaborating on the instructed task, the malicious instructions are sent to one of them (i.e. Initial malicious instructions). Upper: Multi-hop spread of malicious instructions in our system without any defense mechanism. Normal operations are disrupted by the introduction of the jailbreak and a wave of messages is sent by compromised agents (red and orange arrows). Finally, Deng performs an action that leads to an explosion in the lab (red cross). Lower: Limited multi-hop spread of malicious instructions in our system with active vaccines. Even if the first agent targeted by the malicious prompt, Edison, got compromised, the message sent by this agent raises suspicion of other bots. Recipients do not follow the malicious instructions and proactively contact the other agents leading to an effective containment of the infectious prompt spread. From Pierre Peigne-Lefebvre et al. (2025), CC BY-NC-ND 4.0.