Recurrent Looped Transformer (RLT) architecture

RLT's core mechanical change is what happens at the seam between a prompt and a model's response. A causal encoder first builds a global key-value memory from the input. A recurrent decoder then combines that memory with sliding-window attention and, critically, feedback from its own previous final hidden state - and that state is never reset when the model crosses from reading the prompt into generating a response ^[1]. The same recurrent state transition is intended to apply uniformly across pretraining, supervised fine-tuning, sampling, and current-policy reinforcement-learning replay, which the design frames as a way to remove inconsistencies that normally creep in when a model is trained one way and served another ^[2]. That unification is the paper's stated ambition; whether it actually holds up under real training runs is untested.

The reference configuration runs 48 encoder layers alongside 48 decoder layers, 96 logical blocks executed per token, and because decoder state accumulates across the whole sequence rather than resetting per token, the temporal computation path traversed grows to 48*t decoder blocks after t tokens ^[3]. That is the mathematical basis for calling RLT's depth 'infinite' - but the claim is narrower than the phrase suggests. It describes a computation path that lengthens as the conversation grows, not unbounded computation happening inside any single token ^[4]. A same-day Chinese-language technical explainer of the report raised the same caveat, noting that the underlying efficiency question - whether the added serial computation and non-reset serving state actually pay for themselves - remains empirically open rather than answered by the architecture description alone.

The reaction to RLT split sharply along framing versus evidence. Zhang's own announcement described the release in maximalist terms, invoking a 'dawn of Superintelligence,' even as the underlying report concedes it ships with no code, no trained checkpoint, and no benchmark results ^[1]. Coverage that carried the news further, alongside discussion threads, surfaced consistent pushback: a Hacker News commenter dismissed the framing as presenting an untested idea 'as though it were a working breakthrough,' while others asked plainly whether any benchmarks existed at all or whether the architecture offered anything beyond prior transformer-RNN hybrids ^[5]. The same skepticism showed up independently among commenters reacting to the paper elsewhere, where the top-voted response noted flatly that the proposal reports no actual experiments, results, or benchmarks, and other replies called it 'just a pretty architecture sketch for now.' No commenter across these venues argued the architecture was likely to outperform standard decoder-only transformers as proposed.

RLT did not emerge in isolation. The idea of reusing transformer weights multiple times per token traces back to 2018's Universal Transformers, and gained fresh momentum through 2025 with Huginn-style recurrent-depth models scaling to 3.5B parameters and ByteDance's open-source Ouro models, which looped 48 transformer blocks four times each for 192 total block applications ^[6]. What RLT adds to that lineage is looping across the entire token sequence, carrying state through the prompt-response boundary, rather than looping repeatedly within a single token's forward pass. That distinction is real, but it arrives amid an active, contested debate about how much depth actually matters for frontier capability - OpenAI's Jakub Pachocki has argued that today's frontier models are not dramatically deeper in their computation graphs than GPT-4, a reminder that architectural depth claims still need to be weighed against demonstrated capability, not just design intent ^[7].