// MODEL OPTIMIZATION AND PROMPT SYNTAX TERM
Prompt Injection
A security vulnerability where a malicious user manipulates an AI model's behavior by inserting hidden or adversarial instructions into a prompt, overriding its original purpose.
TECHNICAL DEFINITION
Prompt injection is a security vulnerability where an attacker manipulates a large language model's (LLM) behavior or overrides its system instructions by embedding adversarial or conflicting directives within user input, potentially leading to unintended actions, data exfiltration, or harmful content generation.
BACKGROUND
Prompt engineering is the process of structuring natural language inputs to produce specified outputs from a generative artificial intelligence (GenAI) model. Context engineering is the related area of software engineering that focuses on the management of non-prompt and prompt contexts supplied to the GenAI model, such as system instructions, metadata, API tools and tokens.
READ MORE ON WIKIPEDIASYNONYMS & ALIASES
- Jailbreaking
- Adversarial prompting
- Prompt hijacking
- Instruction overriding
USAGE NOTE
Prompt injection is a significant security concern for applications built on LLMs, requiring robust defense mechanisms.
DEVELOPERS
Organizations developing technology related to Prompt Injection.
Develops leading large language models and focuses heavily on AI safety and alignment, including mitigating prompt injection and other adversarial attacks.
Creator of prominent large language models like GPT-4 and ChatGPT, deeply involved in researching and developing defenses against prompt injection to secure their products.
A major AI research lab developing advanced large language models and actively researching and implementing robust defenses against various adversarial attacks, including prompt injection.
Engaged in cutting-edge AI research and product development for Azure AI, including robust security measures for large language models against vulnerabilities like prompt injection.
Specializes in securing large language models, offering platforms and tools specifically designed to detect and prevent prompt injection attacks.
Provides an AI security platform that helps enterprises secure their AI systems, including large language models, against vulnerabilities like prompt injection.
Offers an open-source and enterprise platform for testing and evaluating AI models, including identifying and mitigating security vulnerabilities such as prompt injection in LLMs.
A non-profit foundation that works to improve software security. They publish the OWASP Top 10 for LLM Applications, which prominently features Prompt Injection as a critical vulnerability, influencing industry standards and development.
Focuses on AI safety and security for enterprises using large language models, providing solutions to prevent prompt injection and other adversarial attacks.
An AI security company that provides a platform to detect and prevent adversarial attacks, including prompt injection, on machine learning models across various deployments.