OpenAI's Jalapeño Inference Chip

Jalapeno was designed from a blank sheet as an inference-only ASIC rather than a repurposed training chip. OpenAI has said plainly: "We designed Jalapeno to minimize data movement and communication delays." ^[1]The core idea is a unified prefill-decode architecture - instead of splitting the two phases of generating a response across separate hardware pools, as most GPU inference stacks do, Jalapeno handles both on the same silicon, with localized HBM sitting close to the compute that needs it. Each package carries six HBM4 stacks totaling 216 GiB of memory and 15.4 TB/s of bandwidth, and the design scales from 128-accelerator racks up to 2,048-chip pods delivering roughly 27 EFLOP/s of aggregate compute. ^[2]Getting there fast mattered too: OpenAI used its own models, including Codex, to help write and tune the chip's Gluon-language kernels - some hand-tuned to around 3,000 lines with a custom correctness sanitizer - which reportedly cut SIMD area by 8% and matrix-engine area by 10%, part of what let the team go from RTL freeze to tapeout in roughly nine months. ^[3]

The strongest argument for building custom inference silicon isn't raw compute - it's that inference, unlike training, is a recurring operating cost that scales with every user query, which makes power efficiency the thing that actually shows up on the bill. ^[4]Jalapeno's 700W package TDP (with sustained draw often measured lower during testing) compares against 1,200W for Nvidia's GB200 and 1,400W for GB300 and AMD's MI355X, which is where the reported 1.5x-1.9x throughput-per-kilowatt gains come from. Broadcom CEO Hock Tan has gone further, claiming Jalapeno runs inference at roughly half the cost of a typical AI GPU - a claim from Broadcom itself, not an independently audited figure. ^[5]Whether that cost advantage holds outside OpenAI's chosen test conditions is exactly the open question the rest of the industry is now arguing about.

OpenAI ran its published Jalapeno numbers through SemiAnalysis's public InferenceX suite, testing its own GPT-OSS-120B alongside DeepSeek R1 and Moonshot AI's Kimi K2.5 against Nvidia's GB200 and GB300. ^[6]That is also the crux of the skepticism: SemiAnalysis's own writeup, despite concluding Jalapeno beats Blackwell in nearly every scenario tested, flags that OpenAI supplied all the benchmark data itself and that Blackwell may not even be the right comparison - Jalapeno's HBM4-based design more directly rivals Nvidia's newer Rubin chip, which wasn't tested at all. Community reaction on Reddit landed in roughly the same place: commenters broadly accepted the efficiency numbers as a genuine engineering achievement, but pushed back that this was effectively a vendor demo, run with OpenAI's engineers on-site and OpenAI's chosen models, on a workload (single-turn, fixed-length) that may not represent messier real-world agentic use. Several also noted the timing, landing the week ahead of Nvidia's earnings, as suspicious. On X, semiconductor analyst Dylan Patel - who said he got direct lab access to dissect the chip - called the result "huge news" and unusual for a first-generation chip, lending some independent credibility even as the underlying numbers remain OpenAI-run.

The framing on social media has been unmistakably a rivalry story - OpenAI's own announcement post drew over 14,000 likes, and one widely-shared post claimed Nvidia's Jensen Huang had publicly responded to the comparison - a claim not independently confirmed elsewhere - which is itself a sign of how much this has been read as an OpenAI-versus-Nvidia contest rather than a narrow hardware upgrade. Analysts cited by CNBC said Jalapeno could pressure Nvidia's margins in the fast-growing inference segment and reduce OpenAI's reliance on Nvidia hardware for some workloads. ^[7]But the more understated fact is that OpenAI isn't dropping Nvidia at all: it plans to deploy Jalapeno in its own infrastructure only in very small volumes by the end of 2026, with more meaningful deployment in 2027, while continuing to buy Nvidia hardware in parallel. ^[8]That makes Jalapeno additive capacity and negotiating leverage first, wholesale replacement a much later and less certain story.

Even if Jalapeno's numbers hold up under independent scrutiny, its near-term impact is capped by real supply constraints. TSMC's advanced packaging capacity - needed for Jalapeno's HBM4 stacks, reportedly sourced from Samsung - is sold out through 2026, a bottleneck OpenAI shares with Google and Meta's own custom-silicon efforts. ^[9]That mirrors Google's TPU program, which took roughly a decade of iteration before reaching a dedicated inference-specific chip in Ironwood - Jalapeno is arguably starting from a stronger position given its unusually fast nine-month design cycle, but it is still a first-generation part entering an industry where first-generation chips are rarely competitive out of the gate. OpenAI has said it is already working on Jalapeno's second and third generations, suggesting the real payoff, if there is one, will show up several chip cycles from now rather than in this initial small-volume rollout. ^[10]