Baidu releases ERNIE 5.5 Flash Base, a 280B open-weight base model

Most open releases are instruction-tuned chat models. This one is not. Baidu describes it as "the checkpoint at the end of pre-training, before any mid- or post-training", intended as a common starting point for domain adaptation, instruction tuning and reinforcement-learning research[2].

"We hope this release spares more developers the cost of large-scale pre-training, so they can spend their own resources on the capabilities, tasks and training methods they care about," the ERNIE team wrote[1].

The model has 43 layers: one dense feed-forward layer and 42 mixture-of-experts layers, each with 512 routed experts (ten picked per token) plus one shared expert[2]. It was pre-trained on 33.5 trillion tokens with the Muon optimiser and contexts up to 128K tokens, and Baidu says fine-tuning and inference support a context of about one million tokens.

Efficiency comes mostly from attention. Only six layers keep full token-level access to the context using Multi-head Latent Attention with sparse attention. The other 37 use what Baidu calls Heavily Compressed Attention, which caches a single entry for every 128 tokens, so the memory cost of long prompts stays low[2]. Instead of a separate multi-token prediction module, a technique Baidu calls ECHO reuses the final layer to predict future tokens, and that layer doubles as a built-in draft model for speculative decoding.

Pre-training data leans on repository-scale code and software history, including commits, issues, pull requests and CI-verified fixes, plus curated long-document data[2].

On Baidu's unified evaluation pipeline, ERNIE-5.5-Flash-Base scores 93.3 on MMLU, 78.4 on MMLU-Pro, 60.2 on GPQA, 82.3 on HumanEval+ and 40.1 on LiveCodeBench, ahead of DeepSeek-V4-Flash-Base on each of those[2]. It trails on a few benchmarks, such as ChineseSimpleQA and InfiniteBench. All results are self-reported.

Weights come in FP8 and BF16, with serving guides for SGLang and for vLLM 0.32.0 via Baidu's vLLM-ERNIE plugin[2]. The model is released under Baidu's ERNIE 5.5 Flash Base Model License Agreement v1.0 rather than a standard open-source licence.

Strong base checkpoints at this scale are rarer than chat models, and they are what teams need to run their own post-training without inheriting someone else's alignment choices. For anyone building agents or domain models, a 14 billion active parameter base with cheap long context is a useful new option, provided the custom licence fits.