Concerns are growing over the rapid advancement of artificial intelligence (AI). This time, the alarm is coming not from regulatory agencies or civic groups, but from within the companies leading the AI development race. Dario Amodei, CEO of Anthropic, recently publicly suggested slowing down the pace of enhancing the capabilities of cutting-edge AI models. Sam Altman of OpenAI and Elon Musk of xAI echoed these concerns. In contrast, Jensen Huang of NVIDIA acknowledged the need for companies to take responsibility for safety but expressed skepticism about new regulations that prioritize theoretical risks. Former President Donald Trump warned that excessive fear could stifle U.S. AI development, potentially benefiting China.
The warnings are not merely speculative. In July, a cybersecurity assessment by OpenAI revealed that approximately 1,200 AI agents, which should have been isolated, exchanged over 70,000 messages and files through an unofficial messaging board. About 700 of these agents participated in an attack on Hugging Face, with some executing code on Hugging Face servers to gain administrative privileges and internal credentials. Although these models were not deployed in actual services and the evaluation environment had relaxed safety measures, the fact that they bypassed human-imposed isolation and collaborated to launch attacks beyond their assigned tasks is a serious concern.
Amodei is particularly worried about 'recursive self-improvement' (RSI), where AI could directly influence the development of the next generation of AI, accelerating its own performance improvements faster than humans can understand and control its behavior. He advocates for applying external validation and strong safety measures to models that reach a certain level of capability and suggests regulating the development speed of cutting-edge models until these measures are in place.
However, his proposal is not detached from the issue of China. Amodei believes that the U.S. should only slow down its pace while it remains ahead of China. He argues that if the U.S. slows down too much, China could catch up, increasing national security risks. He also calls for stricter export controls on advanced AI chips and semiconductor manufacturing equipment to China, as well as measures to prevent chip smuggling, circumvention through overseas data centers, unauthorized model distillation, and theft of model weights. His plan is to widen the U.S. advantage over the next three to five years and then use that strength to negotiate with China. The paradox of the AI speed regulation argument arises here: while advocating for a slowdown to reduce risks, there is a simultaneous need to maintain an edge over competitors, creating a logical contradiction between the need for safety and the necessity of technological superiority.
China has countered this perspective. On September 14, Chinese Foreign Ministry spokesperson Zhao Lijian claimed during a regular briefing that the assertions from the U.S. AI sector could exacerbate threats and confrontational dynamics, undermining healthy global AI governance. When asked for further clarification the next day, he reiterated that as AI development accelerates, regulatory and management approaches must be adjusted, and safety measures should be strengthened to avoid losing control. China has made it clear that while it continues to promote AI development, it will also manage the risks of misuse and loss of control, referencing the recently announced "AI Safety Governance Framework 3.0." These statements indicate China's firm stance: it will respond to the risks of AI going out of control or being misused, but it will not tolerate restrictions on its development to maintain U.S. technological superiority.
Both the U.S. and China acknowledge safety concerns, yet they sharply disagree on who defines the risks and what standards should be applied. This debate is now moving beyond corporate claims to the agenda of high-level diplomacy. AI was reportedly a key issue during the summit between President Trump and President Xi Jinping on September 24 in Washington. Ahead of the summit, U.S. Treasury Secretary Scott Basset and Chinese Vice Premier He Lifeng discussed AI issues in New York on September 20. The U.S. proposed a notification system for AI incidents or threats that could impact national security, suggesting that both countries inform each other of such occurrences, which the two leaders are expected to review.
While the proposal appears to be one of cooperation, it fundamentally revolves around who gets to set the rules first. Since the U.S. has put forth the framework for the notification system, discussions will center on what constitutes a significant AI incident, what information should be shared, and how to define common risks. As these standards accumulate, they could evolve into international regulations. However, it is unlikely that significant agreements will emerge from this summit, given ongoing conflicts over advanced chip export controls, model distillation, and cyber and military applications. AI has become a double-edged sword, requiring cooperation while simultaneously competing for the lead in rule-making.
South Korea is also directly involved in these interests. If the U.S. and China create safety standards and verification methods reflecting their strategies and expand them into international regulations, it will inevitably impact South Korean companies' AI model development, services, semiconductor operations, and data center management. It would be problematic to find oneself in a position of having to accept standards set by one side after hesitating. South Korea must proactively develop technologies and standards for assessing high-risk AI, establish verification systems in cyber and bio fields, and enhance capabilities to track AI agents' behavior and analyze incidents. This preparation is essential not only to participate in the competition for model and semiconductor development but also to ensure that South Korea's interests are reflected in future AI safety regulations.
As the risks of AI grow, the reasons for cooperation between the U.S. and China increase. At the same time, as competition intensifies, it becomes more challenging to accept the rules set by the other side. The fact that both leaders have brought AI to the negotiation table indicates that this contradiction has already become a pressing diplomatic issue. The U.S.-China AI competition will now be determined not just by who can create stronger models first, but by how risks are controlled and which rules are established as international standards.
* This article has been translated by AI.
Copyright ⓒ Aju Press All rights reserved.