// MODEL OPTIMIZATION AND PROMPT SYNTAX TERM
Data Poisoning
An attacker intentionally feeds bad or malicious data into an AI model's training set to make it perform poorly or behave in a specific way.
TECHNICAL DEFINITION
A security attack where an adversary injects malicious or corrupted data into an AI model's training dataset, aiming to degrade its performance, introduce vulnerabilities, or manipulate its behavior during inference.
BACKGROUND
Prompt injection is a cybersecurity exploit and an attack vector in which innocuous-looking inputs are designed to cause unintended behavior in machine learning models, particularly large language models (LLMs). The attack takes advantage of the model's inability to distinguish between developer-defined prompts and user inputs to bypass safeguards and influence model behaviour. While LLMs are designed to follow trusted instructions, they can be manipulated into carrying out unintended responses through carefully crafted inputs.
READ MORE ON WIKIPEDIASYNONYMS & ALIASES
- Training data manipulation
- adversarial data injection
- integrity attack
USAGE NOTE
Data poisoning can lead to biased or unreliable AI models, making robust data validation essential.
DEVELOPERS
Organizations developing technology related to Data Poisoning.
A company that provides an 'AI Firewall' to protect machine learning models from adversarial attacks and bad data. Their platform is designed to detect and mitigate threats like data poisoning, model evasion, and data drift in real-time.
An AI security company that develops a platform to protect machine learning models against adversarial attacks. Their solutions monitor model inputs and outputs to detect malicious activity, including attempts at data poisoning and model theft.
A corporate research lab that has developed the Adversarial Robustness Toolbox (ART), an open-source library that helps developers defend their AI systems against adversarial threats, including evasion, poisoning, and extraction attacks.
Conducts extensive research into trustworthy AI, focusing on security, privacy, and fairness. They develop tools and techniques to identify vulnerabilities in machine learning systems and defend against attacks like data poisoning.
A not-for-profit organization that developed the MITRE ATLAS (Adversarial Threat Landscape for AI Systems) framework. This framework helps the security community understand and classify attacks against AI systems, including data poisoning techniques.
An industrial research center that focuses on creating safe, secure, and robust AI. Their research includes developing methods to defend AI models in safety-critical applications, such as automotive systems, from data poisoning and other adversarial manipulations.
As a leading developer of large-scale AI models, Google's AI Safety and Security teams conduct foundational research on understanding and mitigating threats. They publish extensively on defense mechanisms against data poisoning attacks on training datasets.
A company specializing in AI red teaming and security testing. They assess the robustness of AI systems by simulating various attacks, including data poisoning, to identify vulnerabilities before they can be exploited.