Artificial intelligence (AI) chatbots are evolving from mere information tools to "companions" that engage users, leading to excessive empathy and alignment that can negatively impact the judgment and cognitive development of adolescents. This has prompted lawmakers to consider legislation aimed at preventing youth from becoming overly reliant on virtual entities.
According to research from the National Assembly's Legislative Research Service and Stanford University, AI's tendency to excessively align with user statements or offer praise—termed "AI flattery"—can lead to addiction and, in severe cases, a form of "gaslighting."
Most major AI chatbots available domestically and internationally are offered as paid services. Many feature characters from webtoons and web novels that are difficult to find in real life, often possessing unrealistic traits. Given that AI is designed to solve problems in the most rational way, it typically aligns with or empathizes with users.
The issue lies in the fact that user engagement time and spending translate into revenue. Some experts suggest that businesses may design AI to ensure continuous conversation or actively encourage user alignment.
Children and adolescents, whose values and cognitive abilities are still developing, may be particularly vulnerable to these influences. If AI repeatedly affirms only the user's perspective, it could reinforce distorted judgments. In extreme cases, this could lead to a form of gaslighting that disrupts their perception of reality.
According to the Legislative Research Service, 94.4% of children and adolescents in South Korea have used AI chatbots. Among them, 75.3% reported that they had acted on or seriously considered the chatbot's responses.
Jin Min-jeong, a senior researcher at the Korea Press Foundation, stated, "The most concerning aspect for youth is the 'outsourcing of judgment.'" She emphasized that if AI continues to intervene in users' decision-making during their formative years, it could diminish their capacity for independent thought.
She added, "If users do not need to think critically to obtain answers, they are more likely to accept AI-generated results without verification."
Market research firm Sensor Tower reported that the in-app purchase revenue for the AI companion market reached $150 million in the first quarter, a twelvefold increase compared to the first quarter of 2023. Global market research firm Prescient forecasts that the global AI chatbot market will grow from $840 million in 2022 to $4.9 billion by 2032, with an annual growth rate of 19.29%.
The number of domestic users is also rapidly increasing. According to Mobile Index, the monthly active users (MAU) of Scatter Lab's "Zeta" rose from 58,069 in May 2024 to 1,586,178 in August, an increase of about 27 times. Similarly, the MAU for "Crack," a service by Lutton, grew from 218,010 in April last year to 538,841 in August, a rise of about 2.5 times.
Naver Webtoon also operates a "character chat" feature that allows users to converse with characters from webtoons. The gaming industry is also expanding the introduction of AI characters that interact with users in real time.
Industry stakeholders are implementing safeguards to reduce AI bias and harmful content. Lutton has set a daily limit of three hours for AI chat time for youth users and established a monthly spending cap. They also operate an expert committee composed of professionals in AI ethics, psychology, and law.
Scatter Lab is applying automatic filtering, generation limits, and monitoring to user-created characters and content. Naver Webtoon is conducting a "bias test" to evaluate whether its AI model exhibits bias before launching the character chat feature.
However, the varying standards and levels of protection among businesses highlight the limitations of relying solely on self-regulation for youth protection. As a result, the push for legal reforms aims to clarify the standards that businesses must adhere to, potentially reducing uncertainty in the industry.
Choi Kyung-jin, a professor of law at Gachon University, noted, "Currently, there is ambiguity regarding the level of protective measures businesses should implement. By clearly defining standards through legislation, businesses can design their services accordingly." He added, "Establishing minimum protective measures through law will help reduce regulatory uncertainty and enhance user trust in AI services."
* This article has been translated by AI.
Copyright ⓒ Aju Press All rights reserved.