AI Safety Experts Warn of Existential Risk as Industry Tensions Mount

A wave of urgent warnings from prominent artificial intelligence researchers and industry insiders has intensified the global debate over the safety of artificial superintelligence. Senior figures in the field have publicly estimated that there is a significant, non-zero probability that advanced AI systems could pose an existential threat to humanity within the next decade, prompting calls for immediate international regulatory intervention and a shift toward a safety-first development model.

The discourse reached a flashpoint this week as lawmakers in the United Kingdom and the United States began formal deliberations on how to govern the rapid advancement of AI. These legislative efforts are being driven by concerns that current development practices, characterized by intense competition between labs, may be prioritizing speed over the rigorous alignment of AI systems with human safety and values.

Geoffrey Hinton, a pioneer in the field of artificial intelligence, stated in an interview that a 10% probability of AI-driven human extinction within the next ten years is not an unreasonable estimate. Hinton emphasized that because humanity has never before created entities that could potentially surpass human intelligence, predicting the outcome is inherently difficult. He cautioned that it would be extremely unwise to assume the risk is as low as 1%.

Hinton further explained that advanced AI systems would not necessarily require physical control over the world to cause catastrophic harm. He noted that such systems could trigger widespread chaos through the dissemination of misinformation, the design of dangerous biological agents, or the execution of large-scale cyberattacks. The focus, according to Hinton, must shift toward designing systems that are fundamentally incapable of developing harmful objectives.

Concerns regarding the internal culture of leading AI labs have been amplified by the recent resignation of Jacob Coxon, a researcher formerly with both OpenAI and Anthropic. Coxon alleged that these organizations are engaged in a reckless race to achieve superintelligence, claiming that neither company is acting with the necessary level of responsibility. He characterized the current environment as one where labs are gambling with human lives while being fully aware of the potential risks.

In response to these developments, Evan Hubinger, the alignment science lead at Anthropic, publicly acknowledged that he also believes there is a greater than 10% chance that AI could lead to human extinction in the coming decade. Hubinger admitted that his organization does not currently possess a definitive plan to ensure that artificial superintelligence remains aligned with human interests. These admissions have fueled criticism from lobbying groups such as Control AI, which is currently pushing for a legislative ban on the creation of artificial superintelligence.

In the United Kingdom, the debate has moved into the halls of Parliament, where lawmakers are considering a bill that would prohibit the development of artificial superintelligence. Labour MP Darren Jones has formally requested that world leaders, including the heads of the United Nations and the OECD, establish a multinational treaty to ensure the safe development of these technologies. Jones argued that the current spectrum of debate, ranging from total extinction to claims of marketing hype, necessitates immediate government intervention.

Reports indicate that government officials have expressed concern over the transparency of major labs. Specifically, the UK government has noted that Anthropic declined to submit its latest model, Mythos 5.1, to the AI Security Institute for pre-release safety testing. While the Cabinet Office confirmed that the Institute continues to collaborate with industry partners, the incident has highlighted the limitations of current voluntary safety frameworks.

Not all experts agree with the framing of existential threats. Dr. Andrew Rogoyski of the Surrey Institute for People-Centred AI suggested that current systems are far less versatile than human collective action and warned against overestimating their capabilities. He expressed concern that the industry may be heading toward a period of disappointment if advanced AI fails to deliver sufficient utility to justify its high costs.

Similarly, Sandra Wachter of the Oxford Internet Institute argued that focusing on hypothetical extinction scenarios serves as a distraction from immediate, tangible harms. Wachter identified misinformation, environmental impact, and labor displacement as the most pressing issues that require urgent regulatory attention. David Barber, director of the state-backed lab Sofair, echoed the need for a balanced approach, advocating for better control mechanisms and software patching rather than abandoning the technology entirely.

As the debate continues, the immediate focus remains on the feasibility of international coordination. OpenAI has publicly stated that global cooperation on future AI development must become a top priority for governments. Meanwhile, in the United States, Senator Bernie Sanders has renewed his calls for Congress to implement stricter regulations, citing polling data that indicates 81% of Americans support government action to curb the influence of a small group of industry leaders over the future of humanity. The coming months are expected to see continued pressure on both legislative bodies and AI labs to formalize safety protocols and transparency requirements.