Data poisoning is a poisoning attack in which an adversary controls part of the training data, according to the National Institute of Standards and Technology. NIST places it inside a wider class of adversarial attacks in which an adversary interferes with a model during its training stage.
The term covers manipulation of the training data itself. Modifying the training process is a separate entry in the same taxonomy, which NIST calls model poisoning, a poisoning attack that operates through model control.
Key Takeaways
- NIST defines data poisoning as a poisoning attack in which an adversary controls part of the training data, in its adversarial machine learning taxonomy AI 100-2e2025.
- GenAI foundation model developers commonly scrape data from a wide range of sources, per NIST. The scale and diversity of those sources provide a large potential attack surface. Attackers may seek to insert their own data points into it.
- Article 15(5) of the Artificial Intelligence Act names data poisoning in its operative text. It asks high-risk AI systems to include, where appropriate, measures to prevent, detect, respond to, resolve and control attacks trying to manipulate the training dataset.
- Researchers pretrained models from 600 million to 13 billion parameters. They found that 250 poisoned documents similarly compromise models across all model and dataset sizes tested. That held despite the largest models training on more than 20 times more clean data.
- Data filtering can also attempt to remove poisoned samples, according to NIST. Detecting poisoned data within a large training corpus may be very difficult.
How Does Data Poisoning Work?
Three conditions have to line up. The attacker needs data the model will read. Then comes a way to place chosen samples into that data, and something later that makes the planted behavior surface.
1. Find Data the Model Will Read
The performance of GenAI text-to-image and text-to-text models has been found to scale with dataset size. NIST cites work by Hoffmann and colleagues suggesting that compute-optimally training a 520 billion parameter model may require 11 trillion tokens of training data.
Corpora at that scale aren’t assembled by hand. The machine learning adoption and scale data shows how quickly training runs outgrew hand-curated sets, and curation is exactly what a scraped corpus trades away.
2. Insert the Poisoned Samples
Dataset publishers may provide a list of URLs to constitute a training dataset, according to NIST. Attackers may be able to purchase some of those domains. Site content then gets replaced with their own malicious content.
Beyond the vast quantities of pre-training data, data poisoning attacks may also affect other stages of the LLM training pipeline. Those stages include instruction tuning and reinforcement learning from human feedback. Both may intentionally source data from a large number of human participants.
OWASP splits the same surface across pre-training, fine-tuning, and embedding stages. It classifies data poisoning as an integrity attack because tampering with training data impacts the model’s ability to make accurate predictions.
3. Wait for the Trigger
Poisoning may allow for the implementation of a backdoor, per OWASP. Such backdoors may leave the model’s behavior untouched until a certain trigger causes it to change. In effect, that creates the opportunity for a model to become a sleeper agent.
NIST’s glossary supplies the canonical worked example from computer vision. An adversary could poison a model such that the insertion of a square of white pixels induces a desired target label.
Two comparisons make the shape concrete, and both should feel familiar. A recipe book gets one ingredient quietly edited on a single page. Every cook who follows that page then produces the same wrong dish, without doing anything wrong themselves.
The other comparison sits closer to the plumbing. Contamination introduced at the mill travels to every kitchen downstream. The restaurant serving the meal never had a chance to catch it at its own door.
| Stage | What the attacker supplies | What the source says about the stage | Source |
|---|---|---|---|
| Pre-training | Content on the domains that serve dataset URLs | Scraping from a wide range of sources provides a large potential attack surface | NIST |
| Instruction tuning | Instruction and response pairs inside the tuning set | Poisoning may also affect other stages of the LLM training pipeline | NIST |
| Reinforcement learning from human feedback | Feedback submitted among many participants | These stages may intentionally source data from a large number of human participants | NIST |
| Embedding | Text that gets converted into numerical vectors | Poisoning can target the embedding stage, where text becomes numerical vectors | OWASP |
Sources: NIST AI 100-2e2025, OWASP GenAI Security Project
Why Does Data Poisoning Matter?
The attack has a name in law, which is rare for a technique this young. Article 15(5) of the Artificial Intelligence Act requires high-risk AI systems to be resilient against attempts by unauthorized third parties. Those attempts cover altering a system’s use, outputs, or performance by exploiting system vulnerabilities.
Three primary sources bracket the problem, and they disagree about how tractable it is.
- The technical solutions addressing AI-specific vulnerabilities shall include, where appropriate, measures to prevent, detect, respond to, resolve, and control attacks trying to manipulate the training data set. They shall also be appropriate to the relevant circumstances and the risks.
- Cyberattacks against AI systems can leverage AI-specific assets, such as training data sets, according to the recital that accompanies the article.
- NIST advises red teams to assess resilience against ML attacks, in the Generative Artificial Intelligence Profile of its AI Risk Management Framework. The named list runs adversarial examples, data poisoning, membership inference, model extraction, and sponge examples.
Scope matters before any of that gets quoted at a vendor. The duty attaches to high-risk AI systems rather than to every model a company runs.
The statute also hedges twice, asking for measures where appropriate and making them proportionate to the circumstances and the risks. What NIST contributes is a recommendation inside a voluntary framework.
The same NIST publication that supplies the taxonomy reads differently on detection. Data filtering can also attempt to remove poisoned samples, though detecting poisoned data within a large training corpus may be very difficult.
Set those documents beside each other, and the gap becomes the story. A regulator has written a detection duty into the operative text of a statute. The standards body that wrote the taxonomy says detection inside a large corpus may be very difficult. Both sentences are primary-source text, and they point in opposite directions.
The resolution that holds up is provenance rather than inspection. A buyer can confirm that the dataset they downloaded matches the hash the publisher posted. Nobody reads a corpus hunting for a handful of documents.
That pushes the useful comparison up a level, from models to the suppliers who assembled their training data. The AI model security comparison data sets that column beside the capability numbers, which is where a procurement question gets settled.
Types of Data Poisoning Attacks
NIST’s taxonomy puts data poisoning and model poisoning under a shared parent. Poisoning attacks are adversarial attacks in which an adversary interferes with a model during its training stage. That happens either by inserting malicious training data or by modifying the training process itself.
A backdoor poisoning attack makes a model perform an adversary-selected behavior. The trigger is any input that follows a particular backdoor pattern. NIST defines that pattern as a transformation or insertion applied to a data sample.
Two named techniques target web-scale corpora specifically, and both come from the same research paper.
Split-view poisoning exploits the mutable nature of internet content. The result is that a dataset annotator’s initial view of the dataset differs from the view downloaded by subsequent clients. Frontrunning poisoning targets web-scale datasets that periodically snapshot crowd-sourced content, such as Wikipedia. There, an attacker only needs a time-limited window to inject malicious examples.
The boundary readers blur most often sits at the far end of the pipeline. Poisoning reaches the training set before a model exists, while prompt injection works on a model that is already running. The inference-time prompt attack data covers that side of the split.
| Attack | What the adversary controls | What a downstream reader sees | Source |
|---|---|---|---|
| Data poisoning | Part of the training data | A model trained on data the attacker partly chose | NIST |
| Model poisoning | The training process, through model control | A supplied model rather than supplied data | NIST |
| Backdoor poisoning | Training data plus a chosen backdoor pattern | Ordinary behavior until an input carries the pattern | NIST |
| Split-view poisoning | Content served at dataset URLs after annotation | An annotator’s view differing from the downloaded view | Carlini et al. |
| Frontrunning poisoning | A time-limited window before a snapshot | Crowd-sourced content captured mid-edit | Carlini et al. |
Sources: NIST AI 100-2e2025, Carlini et al., IEEE Symposium on Security and Privacy
How Much Poisoned Data Does an Attack Need?
Existing work has studied pretraining poisoning assuming adversaries control a percentage of the training corpus. For large models, even small percentages translate to impractically large amounts of data.
The largest pretraining poisoning experiments to date demonstrate that poisoning attacks instead require a near-constant number of documents regardless of dataset size.
The team pretrained models from 600 million to 13 billion parameters. Their Chinchilla-optimal datasets ran 6 billion to 260 billion tokens. They found that 250 poisoned documents similarly compromise models across all model and dataset sizes. That held despite the largest models training on more than 20 times more clean data.
Those results suggest that injecting backdoors through data poisoning may be easier for large models than previously believed. The number of poisons required does not scale up with model size.
Scope carries as much weight here as the headline figure. The authors pretrained those models themselves for the experiment, so the finding describes a controlled document count rather than a shipping model that was poisoned.
A separate team reported that their two dataset poisoning attacks are immediately practical and could poison 10 popular datasets. They showed how they could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD.
The two studies answer the same question in different units, and the unit is what changes a buying decision.
- A percentage of the corpus grows more expensive to hold as the corpus grows.
- A fixed document count does not move with corpus size at all.
- Both figures sit in public research, and neither paper states the contrast.
What Verification and Filtering Achieve
For data poisoning with web-scale data dependencies, NIST points to verifying web downloads as a basic integrity check. The point is to ensure that domain hijacking has not injected new sources of data into the training dataset. The mechanism is stated plainly: the provider publishes cryptographic hashes, and the downloader verifies the training data.
That control does one job well. It tells a downloader whether the data received matches what the publisher intended, which makes silent substitution at the domain layer harder to hide.
Trade-offs and Risks
- A backdoor may leave the model’s behavior untouched until a certain trigger causes it to change. OWASP says that may make such changes hard to test for and detect.
- Hash verification says nothing about whether the publisher’s own corpus was already poisoned before it was hashed. It confirms delivery, not content.
- Filtering and inspection both scale with corpus size, while the number of documents an attacker needs does not.
A clean fine-tune is not a reset: Researchers have identified attacks in which malicious backdoors in pre-trained models can persist, according to NIST. The persistence holds even after downstream users fine-tune the model for their own use. It also survives additional safety training measures.
Read as a procurement consequence, that item reorders the questions. Training on fresh clean data doesn’t clear a problem that arrived with the weights. The burden moves onto the supplier and the corpus they used.
Government Guidance on Securing Training Data
CISA, the National Security Agency, the Federal Bureau of Investigation, and international partners released a joint Cybersecurity Information Sheet on AI Data Security. It covers best practices for securing data used to train and operate AI systems.
The sheet highlights the critical role of data security in ensuring the accuracy, integrity, and trustworthiness of AI outcomes. It outlines key risks that may arise from data security and integrity issues. Those risks span all phases of the AI lifecycle, from development and testing to deployment and operation.
The advisory carries a May 2025 date and the signatures of three US agencies plus international partners. For a governance reader, that combination is the useful part, because the surface moved out of research papers and into multi-government guidance.
Web-Scale Datasets as a Named Target
LAION-400M and COYO-700M are the corpora named in the research. The same paper records that its authors notified the maintainers of each affected dataset and recommended several low-overhead defenses.
Picture the downstream case. A team fine-tunes an open-weights model on a scraped corpus, runs its own safety training, and ships. Nothing in that workflow inspects the pre-training data, and nothing in it would surface a backdoor that arrived before the team started.
A Named Legal Requirement
Article 15(5) names data poisoning and model poisoning in the same sentence. The list covers attacks that try to manipulate the training dataset and attacks on pre-trained components used in training. Adversarial examples, confidentiality attacks, and model flaws sit in the same clause.
Whether a system falls inside that scope is a jurisdiction question before it becomes a security question. The AI regulation status by jurisdiction tracks where each rulebook currently stands.
What Is the Difference Between Data Poisoning and Prompt Injection?
Timing separates them. Data poisoning changes the training data before a model exists, so the flaw ships inside the weights every user receives.
Prompt injection manipulates a model that is already deployed, one session at a time.
NIST keeps the two in separate categories inside the same red-teaming recommendation, listing prompt injection among GAI attacks and data poisoning among ML attacks.
The owners differ too. Poisoning is a supply-chain question for whoever assembled the corpus, while injection is an application-architecture question for whoever wired the model to tools and data.
Can Data Poisoning Be Detected?
Sometimes, and not reliably at scale. NIST describes data filtering as something that can also attempt to remove poisoned samples. It also states that detecting poisoned data within a large training corpus may be very difficult.
The more useful question changes what gets checked. Verifying a download against a published hash returns a yes-or-no answer about provenance, and that’s a different thing from knowing a corpus is clean. A team that can’t answer the provenance question has no answer to the detection question either.
Conclusion
Across the model sizes tested, 250 poisoned documents similarly compromised models despite the largest training on more than 20 times more clean data. That result turns poisoning into a supply-chain question rather than a scale question, because growing the corpus does not dilute a fixed document count.
The checkable control currently points in one direction. Published cryptographic hashes and downloader verification give a buyer a yes-or-no answer about provenance. Reading a corpus for a handful of documents doesn’t. Neither the standards work nor the joint government guidance closes that gap yet, and the detection duty is already written into law.
