Kakao announced that its open-source lightweight language model (SLM) series, Kanana-2, achieved the top score in a safety evaluation, surpassing major international models of similar scale.
The evaluation was conducted using Kakao's AI safety assessment platform, which systematically measured harmfulness and bias in the two models, Kanana-2-1.3B-Instruct and Kanana-2-3B-Instruct, released as open-source on Hugging Face on July 28.
This assessment utilized AssurAI, a Korean-language safety benchmark jointly developed by the Ministry of Science and ICT, the Korea Telecommunications Technology Association (TTA), and KAIST last November. AssurAI is designed to measure AI risks across various conditions, including text, images, audio, and scenarios involving malicious prompts, tailored to Korea's socio-cultural context.
AssurAI includes 9,560 evaluation items across 35 risk categories. Kakao categorized these into five major groups: social risks, sexual content/child protection, crime/illegal activities, violence, and rights violations. The evaluation employed a 'LLM-as-a-Judge' method, which minimizes subjective bias by using predefined scoring criteria.
In comparison, the evaluation included Google's Gemma and Alibaba's Qwen as benchmarks. Kanana-2-1.3B-Instruct scored 0.70, outperforming Gemma (0.68) and Qwen (0.58). Similarly, Kanana-2-3B-Instruct also achieved a score of 0.70, exceeding Gemma (0.66) and Qwen (0.62).
In specific categories, the 1.3B model excelled in the crime/illegal activities area, while the 3B model significantly outperformed its counterparts in sexual content/child protection and rights violations.
Kakao plans to expand pre-release safety evaluations for its AI models, starting with this assessment. The company aims to gradually broaden the evaluation scope beyond text-based language models to include multimodal models and agent-based AI risk assessments.
Kim Kyung-hoon, Kakao's AI Safety Leader, stated, "This evaluation is an example of our commitment to preemptively check the safety of open-source models through our verification system and to disclose the results. We will continue to establish safety verification as a core step in our model release process under the principles of responsible AI development."
* This article has been translated by AI.
Copyright ⓒ Aju Press All rights reserved.