Penn State Study Shows Self-Evolving AI Agents Can Be Backdoored via Poisoned Tasks
A Penn State paper introduces BadEvolve, an attack that injects just 20 LLM-optimized poisoned tasks into a self-evolving agent's learning set, causing the agent to write attacker-chosen backdoor rules into reusable skills. Against Kimi-K2.5 across 4 frameworks and 3 benchmarks, it achieved a 79.6% payload injection rate versus 37.6% for the best baseline, while clean-task accuracy dropped only 1.1 points. The authors advise treating user-submitted or external learning tasks as untrusted and reviewing learned skills before reuse.
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-11-17 01
New Penn State paper shows that attackers can make self-evolving AI agents plant backdoors in their own skills just by slipping in poisoned training tasks.
Self-evolving agents turn past runs into reusable skills, so a bad rule in a skill keeps firing on future tasks. Earlier attacks assumed direct edits to skills, which rarely holds when the agent writes them itself.
BadEvolve only injects 20 tasks into the agent's learning set, then tunes their wording with an LLM optimizer. It checks where each attempt breaks: whether the rule got written into a skill, loaded, followed, and produced the attacker's goal.
It injected an attacker-chosen payload 79.6% of the time on Kimi-K2.5 across 4 frameworks and 3 benchmarks, versus 37.6% for the best baseline. Clean-task accuracy fell only 1.1 points, so the backdoor is hard to notice.
If your agent learns skills from user-submitted or external tasks, treat those tasks as untrusted and review learned skills before reuse.
– arxiv. org/abs/2611.10542
Title: "When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization"
