Under the Hood: The Architecture Behind Kimi K3's Efficiency
Kimi K3's headline number is 2.8 trillion parameters, but the number that actually explains its performance is 104 billion - the count of parameters active on any given token, since the Mixture-of-Experts router fires only 16 of the model's 896 experts at a time. That sparsity is what makes a model nearly three times the raw size of Kimi K2 remain servable by third-party inference providers on release day, rather than confined to Moonshot's own data centers.
The efficiency doesn't stop at sparsity. Moonshot paired the MoE design with two new architectural components: Kimi Delta Attention (KDA), a hybrid linear attention mechanism that swaps standard quadratic attention for a cheaper long-context alternative, and Attention Residuals (AttnRes), which appear designed to preserve accuracy that hybrid attention schemes typically sacrifice [1]. The published model card lays out the specifics: 93 total layers split between 69 KDA layers and 24 gated multi-head latent attention layers, a 160K-token vocabulary, MXFP4 weight quantization paired with MXFP8 activations for inference efficiency, and a 401-million-parameter MoonViT-V2 vision encoder built in for native image understanding rather than bolted on afterward [2].
The architecture bet paid off on benchmarks developers actually care about. K3 took the top spot on Arena's Frontend Code Arena, ahead of Anthropic's own Fable 5 [3], and landed third globally on the Artificial Analysis Intelligence Index, trailing only GPT-5.6 Sol by a narrow margin [4]. For a model published as free, downloadable weights rather than a metered API, ranking within striking distance of the two best closed models on earth is the detail that turned a routine open-source release into an industry event.



