ETRI Launches Ambitious Project for a 100B-Scale AI Model

The Electronics and Telecommunications Research Institute (ETRI), a leading government-funded research entity in South Korea, has initiated the development of a large-scale artificial intelligence (AI) model comprising 100 billion parameters. This endeavor follows ETRI’s selection as one of five national research teams within the NC AI Consortium, a project promoted by the Ministry of Science and ICT. ETRI commenced its full-scale AI research in August 2023, marking a significant step towards enhancing the nation’s AI capabilities.

ETRI’s participation in this consortium is vital for the “Development of a Scalable Multimodal Generative Foundation Model for Industrial AI Transition.” This project aims to create large-scale AI models applicable in various industries, focusing on integrating and processing data across multiple modalities, including text, voice, and images. By doing so, ETRI expects to drive innovation in key sectors such as manufacturing, healthcare, education, and culture.

Research Infrastructure and Goals

To support this initiative, ETRI has established a robust research infrastructure, leveraging data from the National Information Society Agency (NIA) and GPU resources from the National IT Industry Promotion Agency (NIPA). This foundation enables the smooth pre-training of the 100B-scale model, positioning ETRI to contribute to South Korea’s technological independence in the realm of ultra-large AI models.

The project aligns with national strategies aimed at bolstering AI competitiveness, emphasizing the independent development of Large Language Models (LLMs) and multimodal AI technologies. These foundational technologies are anticipated to benefit various sectors, including industry, public services, and academia.

ETRI’s Intelligence Information Research Division is actively incorporating existing core technologies from its national projects into this research. For instance, the language model “Eagle” is being utilized to enhance conceptual understanding and inference capabilities. Additionally, the use of sparse adapter-based persistent learning technology aims to improve the model’s efficiency and relevance.

Expanding Multimodal Capabilities

The Embodied Integrated Intelligence Research Section is working on multimodal AI models focused on audio and video. Building on previous research, including “Technology for Assessing Degenerative Brain Function Decline” and “Multi-Speaker Conversation Modeling Technology,” this section plans to expand its work into general-purpose foundational models.

Moreover, the Visual Intelligence Research Section is researching ways to enhance the performance of vision-language fusion generation and ensure AI safety. This builds on technologies developed through the text-based image generation model “KOALA” and the visual language question-answering model “Ko-LLaVA.”

Oh-woog Kwon, assistant vice president of ETRI’s Intelligence Information Research Division and project manager, emphasized the importance of ETRI’s role as the sole government-funded research institute representing the nation. He remarked, “ETRI has overcome initial resource acquisition challenges and has been steadily pre-training a 100B-scale model. Through close collaboration with NC AI, we aim to lead the industrial AI transformation by developing a scalable and highly reliable multimodal foundation model.”

The roadmap set by ETRI connects technology development, large-scale pre-training, and industrial implementation. With the backing of both the NIA and NIPA, the project is poised to yield significant advancements in South Korea’s quest for independent, ultra-large-scale AI technology capabilities.

As ETRI progresses with the 100B-scale model, its efforts are expected to play a crucial role in establishing South Korea as a global leader in AI technology. By spearheading innovation and facilitating the intelligent transformation of industries nationwide, ETRI aims to transform the landscape of the AI ecosystem in the region.