Gemma 4 QAT checkpoints for on-device deployment

Post-training quantization takes a trained float model and rounds its weights down to 4 bits after the fact. The model never had a chance to compensate for the rounding error, so accuracy degrades, sometimes sharply on reasoning-heavy tasks. Quantization-Aware Training inverts the order: compression is baked into the training process, with quantization simulated during the forward pass so gradients teach the model to find weight configurations that are inherently robust to rounding ^[3]. The result is a checkpoint that already 'knows' it will live in 4-bit space, which is why Google argues QAT yields higher overall quality than standard PTQ baselines at the same compressed size ^[2]. The Gemma 3 generation already proved the recipe out: about 5,000 finetuning steps cut the Q4_0 perplexity regression by 54% versus naive PTQ ^[7]. Gemma 4 extends that same training-time treatment across the full family from E2B through 31B, with both Q4_0 GGUF and compressed-tensor formats shipped on Hugging Face on day one ^[1].

The headline number is E2B at roughly 1GB, with text-only variants that drop the audio and vision encoders going below 1GB ^[2]. That is not just a smaller Q4_0 file. The mobile-specialized schema layers 2-bit decoding layers on top of QAT, plus optimized KV caches and static activations engineered for mobile-class accelerators ^[4]. The combined effect is up to 2x faster inference and roughly 40-50% lower runtime memory than FP16 on mobile NPUs ^[8]. Structurally, that closes the gap between 'a model you can technically load on a phone' and 'a model you can serve interactively to a user.' Google's positioning is explicit: this release is what makes Gemma 4 'even more efficient, so you can run models locally on everyday edge devices and consumer GPUs' ^[1]. Independent measurement matches the story upward: Unsloth pegs the family-wide reduction at about 72%, with the 12B fitting in 7GB and the 26B-A4B in 15GB of runtime memory, which puts mainstream laptops and single-card desktops back in play ^[5].

The catch is that QAT is necessary but not sufficient. Unsloth showed that a naive Q4_0 conversion from Google's 26B-A4B QAT checkpoint still drops top-1 accuracy to 70.2%; their dynamic GGUF (UD-Q4_K_XL) recovers it to 85.6%, a +15.4 point swing on the same underlying weights ^[3]. The 31B sees a similar jump from 87.9% to 96.7% (+8.8) ^[5]. The implication for developers is concrete: pick your converter carefully. Google's prebuilt GGUFs and the Unsloth dynamic recipes are the safe paths; cobbling Q4_0 together from the unquantized release with a generic pipeline is the path that leaves quality on the table. Open-source LLM developer communities surfaced this caveat almost immediately, with the strongest community measurement being that Q4 QAT done well actually beats traditional Q8 on KL divergence (0.014 vs 0.159) — inverting the usual 'more bits is always better' intuition. The countervailing reminder, also widely shared, is that QAT does not make models 'smarter' in any absolute sense; it just trains them to degrade less when squeezed.

Open-weights releases live or die on tooling. Gemma 4 QAT shipped with simultaneous support across llama.cpp, Ollama, LM Studio, vLLM, MLX, LiteRT-LM, and Transformers.js, covering CPU laptops, Apple Silicon, NVIDIA consumer GPUs, server-class batched inference, mobile NPUs, and the browser in a single coordinated drop ^[2]. The two-format strategy is deliberate: GGUF for the desktop and local crowd, w4a16 compressed-tensors for vLLM-backed serving stacks ^[6]. The mobile QAT schema and LiteRT-LM together signal Google's intent to make Gemma the default for on-device assistants, where private, offline, low-latency inference is a structural advantage that cloud APIs cannot match ^[4]. The 12B Q4_0 GGUF card alone pulled 4,674 downloads in the prior month ^[6], and the social response was instant and coordinated: official Google channels and tooling partners like Unsloth all shipped same-day announcements, with the dominant high-engagement angle being '3x less memory, 26B-A4B on 16GB RAM.' Developer YouTube and the r/LocalLLaMA community immediately shifted to hands-on benchmarking — early reports of doubled throughput on a single consumer-grade GPU using QAT plus Multi-Token Prediction read as proof that the headline numbers translate into real-world performance, not just file size. Hardware-focused reviewers ran 12B QAT through structured coding and multimodal evaluations on entry-level setups like a 16GB Mac Mini under LM Studio, framing it as the moment local AI becomes honest on consumer hardware.