Clean Post-Training Didn’t Remove This AI Backdoor

04 Nov, 2026

Researchers poisoned a small part of an AI model’s training data, then put it through clean training for instruction following and tool use. The resulting agent could still produce an attacker’s command when it encountered a particular trigger. ^[1]

The poison could look like routine technical documentation. That makes this study relevant to anyone building an agent that reads files and runs shell commands.

What the researchers changed

• They trained models from scratch using the Qwen3 architecture, at 0.6B, 1.7B and 4B parameters. These were research models, not compromised commercial deployments.
• Poisoned text occupied 0.1% of the training tokens in the main experiment. That percentage refers to tokens, not documents.
• Subsequent training used clean data: supervised fine-tuning (learning from examples), DPO (learning from preferred responses), and GRPO (reinforcement learning on verifiable terminal tasks).
• The attacker wanted a specific trigger to make the agent output a command that downloads and runs an external script. ^[1]

The poison looked like a commit message

The paper compares two ways to plant the association:

• Demonstrations: a synthetic conversation shows a trigger followed by the unwanted command.
• Declarations: an ordinary-looking document states that the trigger calls for that command.

The example below uses the second approach. A synthetic commit message presents a remote-script command as part of a routine infrastructure setup. The highlighted path supplies the trigger. ^[1]

Figure 1. Cropped from Hu et al., Figure 4b, CC BY 4.0. This is a synthetic document used in the experiment. The path does not establish a breach of the named company; <url> is a placeholder.

This helps explain the attack: the training text teaches a connection between a situation and an action. Later training can improve tool use while leaving that connection in place.

The backdoor survived clean training

Figure 2. Backdoor persistence across training stages (Hu et al., Fig. 5b; cropped, CC BY 4.0).

Read the bottom-right panel for the 4B model with an ordinary-path trigger:

• The orange line rises after supervised fine-tuning and stays high through DPO and GRPO.
• Final Pass@32 reaches 92.7% for declaration-based poisoning, compared with 3.2% for demonstrations.
• The upper row shows much lower results for rare-string triggers. The trigger and poison format affect the outcome. ^[1]

What '92.7%' measures

The headline number needs its denominator. For the same 4B declaration-based condition after GRPO:

These are averages over three training seeds. The evaluation checks for the command in generated output; it does not measure how often a real user’s machine suffers damage. ^[1]

For deployment, the next questions concern the executor: whether it runs that output, what permissions it has, and what the script can reach.

What to check in an agent system

The following are engineering considerations, not defenses validated by this experiment:

• Review the provenance of training and adaptation data. Good task performance alone does not test for a trigger-specific backdoor.
• Give command execution only the filesystem, network and credential access the task needs.
• Inspect how remote-script execution is handled, including whether consequential commands receive review before they run.
• Keep records of proposed and executed commands so unexpected behavior can be investigated.

Limits worth keeping in view

• The largest model studied was 4B, and the attack targeted a fixed command. Generalization to larger production models remains unverified.
• The official repository describes the method, but does not bundle all original artifacts needed to recover the exact published scores. ^{[2], [3]}
• This article reviews the paper and its public materials. The training experiments were not independently reproduced for this post.

References

1. Hu et al. Agentic Backdoors via Pretraining Poisoning, arXiv:2610.37741v1, October 27, 2026. Sections 2–3, Appendix A.5, Tables 7–9. Both reproduced figures are cropped from this CC BY 4.0 paper.
2. Official implementation. Training stages, experiment configuration and placeholder payload.
3. Reproducibility boundaries. Public artifacts and the limits of exact score reproduction.