AA-AgentPerf benchmark and NVIDIA Blackwell efficiency

Most inference benchmarks report tokens per second or cost per million tokens. AA-AgentPerf throws those aside for a deliberately different headline number: Agents per Megawatt, the maximum number of concurrent agents an accelerator platform can serve for each megawatt of power it draws ^[1]. The reasoning is that AI data centers are now bottlenecked by power availability, not rack space or capital, so the right question for a buyer is no longer 'how fast' but 'how many agents can I run inside my power envelope.' What makes the benchmark credible is the workload underneath it: rather than synthetic single-turn prompts, it replays real coding-agent trajectories of up to 200 turns with sequence lengths beyond 100K tokens across 12+ programming languages, scored against multiple service-level tiers ^[1]. Crucially, it is the first inference benchmark to allow the production optimizations labs actually run, KV cache reuse (reusing already-computed attention state instead of recomputing it), speculative decoding (a small draft model proposing tokens a large model verifies in bulk), and disaggregated prefill/decode (splitting the prompt-ingestion and token-generation phases onto different hardware) ^[1]. That makes the results a measure of deployable efficiency, not lab-bench throughput.

NVIDIA's GB300 NVL72 led the launch results, and the mechanism behind its lead is structural, not just a faster chip. The GB300 NVL72 links 72 Blackwell Ultra GPUs and 36 Grace CPUs into a single NVLink fabric with roughly 130 TB/s of bandwidth and ~20-21 TB of HBM3e, presented as one liquid-cooled rack ^[4]. Because the whole rack behaves as one large memory-coherent pool, inference can be aggressively disaggregated, prefill and decode placed on separate GPUs and the giant KV cache shared across the fabric, which Artificial Analysis says yields clear gains in both raw compute and Agents per Megawatt versus single-node deployments ^[1]. The numbers bear this out: the rack-scale GB300 NVL72 is roughly 3x more power-efficient than a single-node B300 (~61,354 vs ~21,053 Agents/MW on the easiest SLO tier) and up to 20x more efficient than the previous-generation H200 ^[1][2]. In other words, the win is not only a generational chip upgrade from Hopper to Blackwell; it is the fabric that lets a long-context agent's working state spread across 72 GPUs without leaving the rack.

On DeepSeek V4 Pro at the easiest SLO tier (20 tokens/s, 10s time-to-first-token), the spread is wide: GB300 NVL72 at ~61,354 Agents/MW, the single-node B300 at ~21,053, AMD's MI355X at ~3,551, and the Hopper-generation H200 at ~2,594 ^[1][2]. That places the GB300 NVL72 roughly 17x ahead of the MI355X and about 24x ahead of the H200 on this tier, with NVIDIA's own '20x more agents per megawatt than H200' headline sitting in the same range ^[2]. The single-node-to-rack-scale jump (B300 to GB300 NVL72) is itself nearly 3x, isolating how much of the gain comes from rack-scale disaggregation rather than the chip alone ^[1]. The shape of the data is the story: this is not a tight race with marginal leads but an order-of-magnitude separation that scales further the harder the service-level tier.

The launch results are a snapshot of immature software, and Artificial Analysis says so explicitly. It cautions that DeepSeek V4 Pro kernel optimizations and config design on AMD systems are in 'relative infancy,' and that it expects significant improvements in AMD performance in the near term ^[1]. That matters because the same production optimizations the benchmark permits, speculative decoding and disaggregated serving, depend on mature, well-tuned kernels, and a two-week-old software stack will leave performance on the table that a hardware spec sheet would not predict. The honest reading is that the GB300 NVL72's lead on rack-scale architecture is real and structural, but the AMD-vs-NVIDIA single-chip gap is partly a software-maturity gap that should narrow as MI355X configurations are tuned ^[1][3]. Buyers comparing platforms on day-one numbers risk locking in a comparison that the next round of kernel work may meaningfully redraw.