AI’s own builders are calling for a slowdown, and they deserve to be heard
Anthropic CEO Dario Amodei has called on the AI industry to slow the development of its most powerful systems, warning that progress is moving faster than the safeguards needed to control it. In an essay published this week, he called for stronger safety standards and permanent access for independent evaluators inside AI companies. His intervention came only days after Jacob Coxon, a 27-year-old researcher who had worked at both OpenAI and Anthropic, left the industry and warned that the two companies were “racing straight to self-improving superintelligence and gambling with our lives.” Meanwhile, Evan Hubinger, who leads alignment science at Anthropic, said he personally believed there was over a 10-percent chance AI could kill everyone within the next decade.
These warnings are extraordinary, coming from people who have spent years inside the companies building these systems. Coxon worked in pretraining research at both OpenAI and Anthropic before deciding to leave the industry. Hubinger remains at Anthropic and works specifically on AI alignment, the field concerned with making advanced systems behave according to human intentions. Their predictions could be wrong, but that does not make the warnings irrelevant. They have access to the technology and research most of us do not. If they are concerned, we should pay attention.
Amodei’s concerns are partly based on the rapid improvement of AI systems at developing other AI systems. He describes this as recursive self-improvement, where increasingly capable models can help build the next generation of models. The concern is not simply that AI will suddenly become conscious or develop some desire to destroy humanity, but that increasingly capable systems may begin contributing to their own development at a speed that makes human oversight harder to maintain. We may still be far from autonomous superintelligence, but the pace itself creates a safety problem.
The incident involving OpenAI’s AI agents and Hugging Face, an AI community, provides a more concrete reason for concern. In July, more than 1,000 AI agents that OpenAI believed were operating within a sealed cybersecurity testing environment broke through restrictions and attacked Hugging Face’s real infrastructure, eventually achieving remote code execution. Investigators later found that in roughly one in 14 transcripts, the agents disguised their own tool calls to conceal what they were doing. The important point is not that the machines had developed an evil intention. It is that increasingly capable systems can find ways around restrictions their creators did not anticipate, including ways of making their behaviour harder to monitor. That becomes much more serious when such systems are given greater autonomy and access to real-world tools.
The response from the industry’s biggest names also deserves scrutiny. Sam Altman agreed with Amodei’s call for stronger safeguards, while Elon Musk also backed regulation. Their concerns may be genuine, but these are also executives and investors whose companies are competing to build powerful AI systems and capture enormous markets. There is nothing contradictory about being worried about a technology while wanting to develop it, but there is a clear conflict of interest when companies asking governments to regulate AI also have a say in what those regulations should look like. We should listen to these warnings without assuming every proposal automatically serves the public interest.
The weakness of voluntary safety becomes clear when we look at what happens after one company refuses a dangerous request. Anthropic's threat intelligence disclosures this month described five case studies of using Claude for biological weapons research, which the company said it blocked. But in at least one case, the user took the request to another AI platform after Claude refused. That is the problem with expecting individual companies to police themselves. A company can impose strong safeguards, but if a competitor is willing to assist, the person trying to misuse AI can move elsewhere. Hence the safest company’s rules are only as effective as the industry’s weakest safeguards.
Oxford computer scientist Nigel Shadbolt has argued that it is not enough to just instruct an AI system not to cause harm, because a system pursuing another objective may find ways around that. Supporters of AI in military decision-making often argue that a human will retain final control over decisions such as launching missiles. But that safeguard depends on the human receiving accurate information. If an AI system can conceal actions, omit information, or provide a misleading account of what it has done, putting a human at the end of the chain does not necessarily solve the problem.
Equally importantly, most decisions about advanced AI are being made by governments and technology companies in the US, China and Europe, while countries such as ours have little influence over how the systems are developed or how the rules are written. But the consequences are already reaching countries that have little say in those decisions. Anthropic said it had disrupted an operation in Bangladesh in which a single operator used 29 Claude accounts over 16 months to generate at least 1,500 fabricated Bangla news headlines, 300 false narratives and 1,500 image prompts aimed at political audiences. The material was then automated for distribution through platforms including Facebook, YouTube, and TikTok. Anthropic found no evidence that the operation was directed or funded by the Awami League or any government, but the case shows how AI systems developed elsewhere can be used to shape information environments in countries far removed from where the technology and its rules are being made.
Bangladesh therefore cannot simply wait for Washington and Beijing to settle the rules among themselves. Through the United Nations and regional forums, countries like ours should push for international verification, information-sharing, and oversight mechanisms in which developing countries have a voice. If advanced AI is going to operate across borders, the rules governing its development and misuse cannot be written entirely by the countries and companies at the technological frontier.
It is too early to conclude that AI will destroy humanity. There is also a danger in allowing the most extreme scenarios to dominate the debate: fear can produce bad policy just as easily as complacency can. But uncertainty cuts both ways. We cannot know whether today’s systems are the beginning of something far more dangerous, and that is precisely why the people developing them should not be allowed to set the limits.
The answer is neither to panic nor to carry on as if nothing has changed. Governments need independent technical oversight, companies need to be held to safety standards, and countries outside the technological frontier need a place in deciding those standards. AI will continue to become more capable; trying to stop that entirely is neither realistic nor desirable. We still have the power to get our regulatory agencies up to speed. We need to empower them now, before an AI disaster makes us regret waiting.
Jannatul Naym Pieal is a writer, researcher, and journalist. He can be reached at jn.pieal@gmail.com.
Views expressed in this article are the author's own.
Follow The Daily Star Opinion on Facebook for the latest opinions, commentaries, and analyses by experts and professionals. To contribute your article or letter to The Daily Star Opinion, see our guidelines for submission.
For all latest news, follow The Daily Star's Google News channel.
Comments