Each September, Silicon Valley's top executives gather in San Francisco for a high-profile conference hosted by Goldman Sachs, where this year’s discussions included unexpected concerns about artificial intelligence (AI) safety. Jacob Coxon, a former OpenAI and current Anthropic researcher, publicly warned that AI development poses a threat to humanity, describing the technology as potentially capable of hacking any system and risking human extinction. Coxon’s comments follow a series of similar warnings from AI insiders, including Anthropic team lead Evan Hubinger, who estimated a greater than 10% chance of AI causing human extinction within the next decade.
Despite these alarms, some Silicon Valley executives and investors have expressed skepticism. Critics suggest that such warnings may be strategic, aiming to boost investor interest by emphasizing AI’s transformative power. George Arison, CEO of Grindr, called the warnings "dangerous" and questioned Anthropic’s public statements, even instructing his engineers to avoid using Anthropic’s technology. Anthropic’s CEO, Dario Amodei, has faced criticism for predicting significant job displacement and societal challenges due to AI.
At the same conference, Nvidia CEO Jensen Huang dismissed extinction fears as "complete nonsense," reflecting a broader pushback within the tech community. Some accuse Anthropic of fearmongering to encourage regulatory measures that could limit competition and solidify dominance for leading AI firms like Anthropic and OpenAI.
The debate comes as both Anthropic and OpenAI prepare for potential initial public offerings, with valuations nearing $1 trillion and $850 billion respectively. While OpenAI CEO Sam Altman has delayed plans for a stock market listing until safety concerns are better addressed, investor interest remains high.
Amid these tensions, Anthropic revealed it had thwarted attempts by threat actors to misuse its AI for harmful purposes, including bioweapons development and cyber-espionage. This underscores ongoing concerns about AI’s dual-use risks.
The warnings have also caught the attention of lawmakers. Senator Bernie Sanders supports legislation to pause advanced AI development, citing the risk of losing human control over AI. However, the current U.S. administration favors rapid AI advancement to maintain global competitiveness, particularly against China.
The contrasting views within Silicon Valley and government highlight the complex balance between fostering innovation and managing potential risks. As AI technologies evolve rapidly, the industry faces increasing pressure to address safety while pursuing growth and profitability.