Superintelligence: Expert Reactions to OpenAI's Vision and Challenges
The race for artificial general intelligence (AGI) and something far beyond it has captured the attention of technologists and the public alike. As a journalist covering the rapidly evolving landscape of AI, I've seen firsthand the ambition and the apprehension that accompany these advancements. OpenAI, a prominent player in this field, finds itself at the forefront of this discussion, grappling not only with the technical hurdles but also with the profound ethical and societal implications of what they are building.
Source: Own illustration
Quick Summary
- OpenAI aims to develop "superintelligence," defined as intellect vastly surpassing human cognitive performance.
- CEO Sam Altman estimates superintelligence could arrive in "a couple thousand days" (5.5 to 11 years).
- OpenAI established a "Superalignment" team, dedicating 20% of its compute power over four years to ensure safe AI development.
- The company acknowledges significant risks, including economic disruption, misuse, and potential disempowerment or extinction of humanity if superintelligence is not properly aligned.
- OpenAI advocates for new industrial policies and democratic decision-making on AI values, moving beyond incremental adjustments.
- Internal challenges, including leadership changes and researcher departures, highlight ongoing tensions regarding safety and rapid development.
The Pursuit of Superintelligence
OpenAI has publicly stated its confidence in building AGI, which it defines as a "highly autonomous system that surpasses humans at most economically valuable work" in its own words. However, the company's ambition extends beyond traditional AGI, focusing on what they term "superintelligence." Sam Altman, OpenAI's CEO, has suggested that superintelligence could arrive in "a couple thousand days." This timeframe, ranging from roughly 5.5 to nearly 11 years, underscores the urgency many in the field feel concerning its development. Superintelligence, as defined by Nick Bostrom in his 2014 book
Superintelligence: Paths, Dangers, Strategies, is "any intellect that greatly surpasses the cognitive performance of humans in virtually all relevant domains."
Source: tedapp.ai
Nick Bostrom’s 2014 book provides a foundational definition of superintelligence, outlining its potential paths, dangers, and strategies.
Altman has called Bostrom's book "the best thing he’s seen on the topic."
OpenAI believes superintelligence could accelerate scientific and medical breakthroughs, significantly boost productivity, and lower costs. The costs associated with building superintelligence are decreasing annually, and the number of entities pursuing its development is growing rapidly. The company acknowledges that the risks of such a powerful technology are significant, including economic disruptions, misuse by malicious actors, misaligned systems, and the concentration of power and wealth. OpenAI believes that superintelligence poses the potential to disempower humanity or even lead to its extinction if not properly aligned.
The Superalignment Initiative
Recognizing these profound risks, OpenAI announced the formation of its "Superalignment" team on July 5, 2023, as detailed in their blog post. Initially led by Ilya Sutskever and Jan Leike, the team’s mission is to solve the problem of governing and controlling AI systems that are far more intelligent than humans. OpenAI has committed 20% of its currently secured compute power over the next four years to address the superalignment challenge. The team's goal is to resolve the core technical problems of superintelligence alignment within four years.
Current AI alignment techniques, which rely on human oversight, will not scale effectively for superintelligence. The Superalignment team's approach involves constructing an automated human-level alignment researcher, developing a scalable training method, validating the model, and stress-testing the entire alignment pipeline. OpenAI published initial findings from its Superalignment team on December 14, 2023, detailing a technique where a less powerful large language model can supervise a more powerful one. For instance, the team used GPT-2 to oversee GPT-4 on various tasks to explore how humans might monitor superhuman models. Results showed that GPT-4, when trained by GPT-2, performed 20% to 70% better on language tasks but worse on chess puzzles. However, AI researcher Thilo Hagendorff raised concerns that GPT-2 might be too "dumb" to serve as an effective teacher.
OpenAI plans to broadly share the results of its Superalignment efforts to contribute to the alignment and safety of non-OpenAI models. To further advance this research, OpenAI has announced a $10 million fund to support researchers working on superalignment.
Governance and Policy in the Age of Intelligence
OpenAI released an article on "Governance of Superintelligence" on May 22, 2023, emphasizing the need to consider the governance of future AI systems far more powerful than AGI. The company suggests three approaches to governing superintelligence: coordination among leading development efforts, an international agency similar to the IAEA for superintelligence endeavors, and the technical capability to make superintelligence safe. OpenAI contends that people worldwide should democratically decide on the values for AI systems.
On April 6, 2026, OpenAI published an article on "Industrial Policy for the Age of Intelligence," presenting ideas for shaping the future with advanced AI. This article suggests that incremental policy adjustments will be insufficient as society moves towards superintelligence. OpenAI offers grants and research awards of up to $100,000 and up to $1 million in API credits for work based on these policy ideas. The company also announced a new workshop in Washington, D.C., in May to host discussions on industrial policy for the age of intelligence.
OpenAI proposes that a new industrial policy is necessary to address the challenges and opportunities presented by superintelligence. This includes giving workers a voice in the AI transition, fostering AI-powered entrepreneurship, and treating access to AI as fundamental to a modern economy. They recommend modernizing the tax base to ensure continued funding for social programs and creating adaptive social safety nets. OpenAI stresses the importance of safeguards before and after deploying AI systems, including real-time monitoring and integration into existing institutions. They propose stricter controls, including audits, for a small number of high-capability models that could pose chemical, biological, radiological, nuclear, or cyber risks.
Internal Dynamics and Departures
The pursuit of superintelligence has not been without its internal challenges at OpenAI. The team responsible for governing superintelligent systems was recently dissolved after its two co-leaders, Ilya Sutskever and Jan Leike, departed the company.
Source: artificial-intelligence.blog
Ilya Sutskever, a co-leader of the Superalignment team, recently departed from OpenAI, marking a significant internal shift.
❝ in recent years, safety culture and processes have taken a backseat to shiny products. ❞
AI Researcher
In June 2024, Ilya Sutskever founded a new company named "Safe Superintelligence."
Sam Altman's brief ousting as CEO in November 2023 was a significant moment, which he characterized as a "major governance failure." An investigation by WilmerHale attributed his removal to a "breakdown of the relationship and loss of trust" between the previous board and Altman. Since late 2023, several top researchers have left OpenAI, including Ilya Sutskever and Alec Radford. These departures highlight the ongoing tensions and differing perspectives within the organization regarding safety, rapid development, and corporate governance. OpenAI currently has three safety bodies: an internal safety advisory group, a board-level safety and security committee, and the deployment safety committee.
OpenAI's Safety Committees
OpenAI currently operates with three distinct safety bodies, reflecting its commitment to addressing the complex risks associated with advanced AI development:
| Committee Name | Primary Function |
|---|---|
| Internal Safety Advisory Group | Provides internal guidance and recommendations on safety protocols and research. |
| Board-Level Safety and Security Committee | Oversees high-level safety and security strategies, reporting directly to the board. |
| Deployment Safety Committee | Focuses on the safety and ethical implications during the deployment of AI systems. |
Conclusion
The debate surrounding superintelligence and its development is complex, encompassing extraordinary potential benefits and equally profound risks. OpenAI, a key architect of this future, continues to navigate these challenges, committing significant resources to alignment research and advocating for forward-looking governance policies. The journey towards superintelligence requires not only technological breakthroughs but also a sustained, global conversation about its ethical implications and how to ensure it ultimately serves humanity's best interests.
Frequently Asked Questions (FAQs)
What is the difference between AGI and Superintelligence?
AGI (Artificial General Intelligence) refers to a highly autonomous system capable of surpassing humans in most economically valuable work. Superintelligence, as defined by Nick Bostrom, is any intellect that vastly exceeds human cognitive performance in virtually all relevant domains, implying a much higher level of capability than AGI.
How quickly does OpenAI expect superintelligence to arrive?
Sam Altman, OpenAI's CEO, has suggested that superintelligence could arrive in "a couple thousand days," which translates to approximately 5.5 to 11 years from his statement.
What is the purpose of OpenAI's Superalignment team?
The Superalignment team's mission is to solve the problem of governing and controlling AI systems that are far more intelligent than humans. They are dedicated to resolving the core technical challenges of superintelligence alignment within four years, ensuring these powerful systems remain safe and beneficial.
What are the main risks associated with superintelligence, according to OpenAI?
OpenAI identifies several significant risks, including economic disruptions, potential misuse by malicious actors, misaligned systems that do not operate in humanity's best interest, and the concentration of power and wealth. They also warn of the potential for superintelligence to disempower humanity or even lead to its extinction if not properly aligned.