FreeToken: UC Berkeley/MIT open-source inference engine runs 753B-parameter MoE models on a single consumer GPU

FreeToken flips the usual assumption that a model must fully fit in GPU memory. It keeps every weight in system RAM and treats the GPU as a global LRU cache that only holds the 'hot' experts - the mixture-of-experts submodules a given token actually routes through ^[2]. Because MoE models activate only a small subset of experts per token, most of a 753-billion-parameter model never has to touch the GPU at all in a given forward pass, which is the core trick that makes multi-hundred-billion-parameter inference possible on an 8-96GB card.

The harder engineering problem is hiding the latency of moving experts in and out of that cache. FreeToken benchmarks PCIe and CPU memory bandwidth once on first run, then dynamically splits cache misses between the PCIe transfer path and CPU-side execution in proportion to whichever is actually faster on that specific machine ^[3]. Prefill and decode are treated as separate bandwidth problems - prefill uses full-layer double-buffered streaming to overlap weight loading with compute, while decode leans on the adaptive LRU cache - and VRAM can be reallocated between the expert cache and the KV cache at runtime without restarting the engine. As one technical walkthrough of the codebase put it, the real bottleneck was never raw compute - it is where and when the expert weights move.

Every piece of press and social coverage around FreeToken's launch tells the same clean story: 2-4x faster than Ollama, a 753-billion-parameter model running at nearly double llama.cpp's speed on a single card. That version of events comes almost entirely from the paper's own benchmark tables and the researchers' own social posts. The most detailed independent scrutiny lives in a long r/LocalLLaMA thread that started as an enthusiastic hands-on report and turned into a pointed methodology fight.

The core objections: no disclosed llama.cpp baseline commit or flags, no documentation of the tuning options that materially change llama.cpp's own offloading performance, single-run numbers with no variance reported, and unresolved credit to prior llama.cpp offloading work. The sharpest claim - that FreeToken's own GitHub pull requests show its bandwidth-based hybrid-execution heuristic performing worse than plain CPU-only execution in some configurations - came from inside the project's own repository, not from an outside benchmark. One independent reproduction did find a real 2.2-3.1x decode speedup over llama.cpp on matched hardware, but with a much slower cold time-to-first-token (21.7 seconds versus 4.7 seconds) - a tradeoff the headline numbers don't mention.

The paper's own framing is straightforward: open-weight frontier models keep getting released, but the infrastructure to serve them still assumes a datacenter ^[1]. That gap has been widening as MoE releases have pushed total parameter counts past what any single consumer GPU could ever hold, even as the number of experts actually needed per token has stayed small.

Traditional CPU-GPU offloading has existed for a while, but it has always been bottlenecked by PCIe bandwidth alone, since it assumes the interconnect is the only variable that matters ^[1]. FreeToken's contribution is measuring the actual bandwidth available on a given machine, both PCIe and host memory, and adapting the execution split accordingly - a small idea that turns out to matter a lot when the difference between a usable chatbot and a frozen one is a few seconds per token.

The 8GB-VRAM headline obscures a real cost: running the largest supported configuration still requires roughly 512GB of system RAM, since that RAM is now doing the job the GPU used to do ^[4]. FreeToken's tiered hardware picture - 35B parameters on an 8GB laptop GPU, 284B on a 32GB desktop card, 753B on a 96GB workstation card - only holds if the corresponding host RAM is also available ^[5].

Community testing surfaced several rough edges the announcement didn't emphasize: the engine currently looks CUDA-only, with no Vulkan path, uncertain support for AMD cards or Apple Silicon, and the best results tied specifically to official NVFP4-quantized checkpoints rather than the GGUF or EXL3 formats most local-LLM users already have on disk. Testers also converged on a practical tip that has nothing to do with the GPU at all: DDR5 memory speed, not just capacity, has an outsized effect on realized throughput, since every cache miss now round-trips through system memory instead of staying on the card.