Motif Technologies has raised objections to the second evaluation results of the independent AI foundation model (AI foundation model) conducted by the Ministry of Science and ICT. The company reversed its initial decision not to contest the results and is now demanding the disclosure of detailed scores and evaluation criteria for each category. Additionally, Motif has stated that it will not participate in the third evaluation, regardless of the outcome of its objections.
On August 27, Motif CEO Lim Seong-hwan issued a statement saying, "The AAII, which received a high score, is not just a simple benchmark but a composite index of nine benchmarks widely used in the global AI industry." He emphasized the need for an explanation of why there was a significant discrepancy between the AAII scores of 47 and 31 and the benchmark conversion scores and expert evaluations.
Previously, Motif had accepted the second evaluation results and stated it would not file an objection. However, after the Ministry's explanation suggested that Motif's model had high benchmark scores but issues in expert and user evaluations, the company changed its stance. CEO Lim noted, "We believe there could be misunderstandings regarding the technical direction and outcomes we pursued, and we request the detailed disclosure of evaluation scores and a re-evaluation."
Motif is requesting that the evaluation results and detailed evaluation content be made available for participating companies to review. CEO Lim stated, "It is important to provide practical feedback and direction so that selected and eliminated companies can prepare better models and strategies in the future."
The company is also seeking clarification on the basis for the scoring and conversion methods used for the benchmark scores. According to Motif, the top model in the AI foundation model evaluation scored 47 points, while the fourth-ranked model scored 31 points, but the gap was reduced to four points when converted to evaluation scores. CEO Lim pointed out that this scoring method may not adequately reflect the performance differences observed in the benchmarks.
Motif has also requested the disclosure of detailed criteria and results for expert evaluations. CEO Lim remarked, "Even considering the technical reports of each model, it is difficult to understand the specific reasoning behind a model with lower benchmark performance ranking first in expert evaluations." He added that during the expert evaluation process, they received written questions that were difficult to understand, such as whether any companies with foreign capital or overseas entities holding voting rights were involved, and emphasized the need for clarification on how such questions were utilized in the evaluation process.
Furthermore, Motif is asking for the methodology, detailed criteria, and results of user evaluations to be made public. The company had previously suggested a blind evaluation method where the developers' identities were not disclosed, arguing that prior knowledge of the developers could influence user evaluations due to brand recognition or biases.
However, Motif has stated that it will not participate in the third stage of the AI foundation model evaluation, regardless of the outcome of its objections. CEO Lim expressed gratitude to the government for providing an opportunity to test their technology through the second evaluation, stating, "In the future, we will strive to create the best frontier model in Korea on our own."
* This article has been translated by AI.
Copyright ⓒ Aju Press All rights reserved.