// MODEL OPTIMIZATION AND PROMPT SYNTAX TERM
Existential Risk
A potential event or outcome that could permanently and drastically harm humanity, possibly leading to its extinction or collapse.
TECHNICAL DEFINITION
An existential risk (X-risk) is a hypothetical event or phenomenon, such as unaligned superintelligent AI, global pandemics, or nuclear war, that could either annihilate intelligent life on Earth or permanently and drastically curtail its future potential, posing a threat to the long-term survival and flourishing of humanity.
BACKGROUND
Existential risk from artificial intelligence, or AI x-risk, refers to the idea that substantial progress in artificial general intelligence (AGI) and artificial superintelligence (ASI) could lead to human extinction or an irreversible global catastrophe.
READ MORE ON WIKIPEDIASYNONYMS & ALIASES
- X-Risk
- Catastrophic Risk
- Global Catastrophe
- Species-Level Threat
USAGE NOTE
Many researchers consider unaligned AGI a significant existential risk that requires proactive mitigation.
DEVELOPERS
Organizations developing technology related to Existential Risk.
A research and deployment company whose mission is to ensure that artificial general intelligence benefits all of humanity. Their work includes a 'Preparedness' team focused on tracking, evaluating, and protecting against catastrophic risks from advanced AI, including existential threats.
A leading AI research lab that has a dedicated AI Safety team working on the long-term challenges of building safe and aligned artificial general intelligence. Their research includes areas like scalable oversight, interpretability, and robustness to mitigate potential large-scale risks.
An AI safety and research company focused on building reliable, interpretable, and steerable AI systems. A core part of their mission is to conduct research into the safety and societal impacts of AI to prevent catastrophic or existential outcomes. They developed the 'Constitutional AI' technique for this purpose.
A non-profit research organization focused on developing formal mathematical theories to ensure that future smarter-than-human AI systems have a positive impact. Their work is centered on foundational alignment problems to prevent existential catastrophe.
A non-profit research organization working to align future machine learning systems with human interests. They focus on theoretical research to solve specific technical alignment problems, such as 'Eliciting Latent Knowledge,' to prevent deception and other catastrophic failures in advanced AI.
A non-profit organization that conducts research and fieldwork to ensure the safe development and deployment of AI. They develop technical evaluations to test for dangerous capabilities and alignment failures in frontier AI models, directly addressing catastrophic and existential risks.
An applied AI alignment research startup working to develop scalable and robust alignment techniques. Their research focuses on understanding and controlling the internal cognition of AI models to prevent unintended and potentially catastrophic consequences.