What is data poisoning?
Data poisoning, also called AI poisoning, is an attack that corrupts the data used to train an AI model in order to degrade its accuracy or insert hidden malicious behavior. By introducing misleading information, altering existing data, or removing key data points, an attacker can cause a model to make wrong decisions or behave in ways that benefit them. It exploits a fundamental truth about machine learning: a model is only as trustworthy as the data it learns from.
Why is data poisoning dangerous?
- It is stealthy: Poisoned data often looks completely legitimate and can slip past standard validation checks.
- It persists: Once a model trains on bad data, the flaw is baked in and can affect the system for its entire operational life.
- It is hard to vet: Modern models, especially LLMs, learn from huge scraped datasets that are nearly impossible to fully inspect.
What are the main types?
- Label flipping: Swapping correct labels for incorrect ones, for example labeling a malignant scan as benign so a diagnostic tool misses it.
- Backdoor attacks: Embedding a hidden trigger so the model behaves normally until it sees a specific signal, then produces the attacker’s chosen output.
- Availability attacks: Flooding the dataset with noisy or contradictory data to degrade overall accuracy and make the model unreliable.
- Clean-label attacks: Subtly manipulating inputs without changing labels, which makes the poison especially hard to detect.
Where does it show up?
- Scraped training data: Attackers seed malicious samples into public, web-scraped datasets that AI companies collect.
- Open-source models and pipelines: Less-restricted access to data and components makes these attacks easier to carry out.
- RAG systems: Researchers have shown that subtle injections into retrieved documents can distort generative AI responses.
How do you defend against it?
- Vet and track data: Use data provenance and integrity checks to verify where training data comes from.
- Audit and monitor: Look for degraded accuracy, unusual outputs, and suspicious data patterns, and compare behavior across model versions.
- Retrain when needed: If poisoning is widespread, retraining on a clean dataset is often more effective than trying to patch the model.
