Baidu Unveils ERNIE 4.5: Open-Source LLMs from 0.3B to 424B Parameters

ERNIE 4.5: A New Era of Open-Source Foundation Models

Baidu has released the ERNIE 4.5 series as open-source, offering a robust collection of foundation models tailored for advanced language understanding, reasoning, and generation. This series includes ten model variants, ranging from compact 0.3 billion parameter dense models to massive Mixture-of-Experts (MoE) architectures with up to 424 billion parameters.

Advanced Model Architecture and Training

The ERNIE 4.5 models improve upon previous versions by integrating both dense and sparse MoE architectures. MoE variants, such as ERNIE 4.5-MoE-3B and ERNIE 4.5-MoE-47B, efficiently activate only a small subset of experts per input token (typically 2 out of 64), maintaining manageable active parameters while preserving performance and generalization.

Training utilizes a combination of supervised fine-tuning, reinforcement learning with human feedback, and contrastive alignment over a vast corpus of 5.6 trillion tokens spanning Chinese and English domains. This enables the models to excel in instruction-following, multi-turn conversations, long-form generation, and reasoning tasks.

Model Variants and Accessibility

The open-source release covers ten variants:

• Dense Models: ERNIE 4.5-0.3B, 0.5B, 1.8B, 4B
• MoE Models: ERNIE 4.5-MoE-3B, 4B, 6B, 15B, 47B, 424B

Notably, the MoE-47B activates only 3 billion parameters during inference, while the 424B-parameter model—the largest from Baidu—uses sparse activation to keep inference efficient. These models support FP16 and INT8 quantization for optimized deployment.

Performance Highlights

ERNIE 4.5 demonstrates leading performance on key benchmarks:

• CMMLU: Achieves state-of-the-art accuracy in Chinese language understanding, surpassing earlier ERNIE versions.
• MMLU: ERNIE 4.5-47B shows competitive results alongside top LLMs such as GPT-4 and Claude.
• Long-form generation: Delivers higher coherence and factual accuracy based on Baidu’s internal metrics.
• Instruction following: Enhanced alignment with user intent and reduced hallucinations due to contrastive fine-tuning.

Practical Applications and Deployment

These models are designed for diverse applications including chatbots, AI assistants with multilingual capabilities, search engines, question answering systems, and content generation with improved factual grounding. Some variants support up to 128K tokens of context, enabling complex tasks involving long documents or conversational sessions. Baidu also hints at compatibility with multimodal extensions in future developments.

Open Access and Future Prospects

By making ERNIE 4.5 openly available on Hugging Face with full documentation and deployment support, Baidu promotes inclusive AI research and innovation worldwide. This release marks a significant milestone in scalable, multilingual, and instruction-aligned large language models.

Explore the paper and models on Hugging Face to dive deeper into this groundbreaking technology.