The artificial intelligence sector is currently engaged in a vigorous discussion about whether AI technology could pose an existential threat to humanity. This debate intensified after AI researcher Jacob Coxon resigned from Anthropic, citing concerns that leading AI companies are "gambling with our lives." Shortly after, Anthropic's alignment lead publicly stated a belief that AI could potentially kill all humans, estimating the probability at over 10% within the next decade.
On a recent episode of TechCrunch's Equity podcast, hosts Kirsten Korosec, Sean O’Kane, and Anthony Ha explored these alarming claims. Ha expressed skepticism about some of the more extreme predictions, noting that assigning precise probabilities to such outcomes often lacks a solid basis. He also highlighted Coxon's resignation as a rare instance of someone acting on their concerns by stepping away from the field.
Korosec suggested that some of the public warnings might serve as a way for companies to demonstrate the advanced capabilities of their AI models, especially as they prepare for public offerings. O’Kane added that these statements could reflect a lack of full control over the technology, pointing to recent reports of AI agents behaving unpredictably within OpenAI’s systems.
With Anthropic approaching its IPO, the conversation has turned to how these existential risk warnings might be reflected in the company’s regulatory filings. O’Kane speculated on whether the company’s legal teams are revising risk disclosures to include the possibility of AI-driven human extinction, and how such admissions might affect investor perceptions.
Korosec noted that, contrary to traditional expectations, acknowledging these risks might not necessarily harm a company’s valuation in the current market environment. Instead, it could paradoxically signal the strength and sophistication of their AI technology.
The discussion also touched on broader concerns about AI governance. Connor Leahy, executive director of the nonprofit ControlAI, emphasized the importance of developing controls to manage AI’s risks. Ha expressed that while the existential threat narrative can overshadow more immediate issues such as labor displacement and environmental impact, it remains crucial to address all dimensions of AI’s societal effects.
Overall, the debate underscores the complexity of balancing transparency about AI’s potential dangers with the commercial and technological ambitions driving the industry forward.