Poisoning, Theft, and Model Integrity

Prompt injection attacks a model at runtime. Two other attack classes target the model further upstream — what it learned, and the value it embodies — and both map cleanly onto the CIA triad from Foundations, now aimed at AI as an asset class.

Data poisoning is an integrity attack on the model's training. Because models learn from data, an attacker who can influence the training or fine-tuning set can corrupt what the model learns: degrade its accuracy broadly, or — more insidiously — plant a backdoor that behaves normally until a specific trigger appears, at which point the model misbehaves on command. Poisoning is especially relevant where training data is scraped from open sources or contributed by many parties, since the attacker only needs to influence a slice of the corpus. The defensive posture is data provenance and validation: know where training data comes from, vet contributions, and monitor model behavior for the anomalies a backdoor would produce.

Model and data theft are confidentiality attacks. A proprietary model is expensive intellectual property, and its training data may contain sensitive or regulated information. Direct theft (exfiltrating the model weights) is the obvious version, but subtler attacks exist: model extraction reconstructs an approximation of a model by querying it heavily, and membership-inference and related attacks can determine whether specific records were in the training set — a privacy breach if that data was sensitive. Some models can even be coaxed to regurgitate memorized training data verbatim.

The reassuring part is that defending AI assets does not require throwing away what you know. It is the familiar discipline applied to a new asset class: protect confidentiality (access control on weights and training data, rate-limiting and monitoring of query interfaces), protect integrity (data provenance, validation, behavioral monitoring), and protect availability (the AI service is a service like any other). The genuinely new work is recognizing the AI system as an asset worth this protection — and the AI-specific attacks, like poisoning and extraction, that standard controls do not by themselves address.