NVIDIA Touts Full-Stack Codesign: Vera Rubin NVL72 Claims 30x Throughput per Dollar
NVIDIA says Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72 with about 3% more HBM per GPU, via full-stack codesign.
NVIDIA says its Vera Rubin NVL72 platform delivers up to 30x higher throughput per dollar than GB300 NVL72 on the SemiAnalysis AgentX DeepSeek V4 Pro workload at 125 tokens per second per user, while HBM capacity per GPU rises only about 3%. The company detailed the claim in a blog post published November 5, framing it as the payoff of “full-stack codesign” — designing GPUs, CPUs, networking, storage and software together rather than simply adding memory.
The post, by Amr Elmeleegy, argues that larger models, longer contexts and agentic applications are increasing GPU memory pressure while memory supply tightens and prices rise, a trend NVIDIA expects to continue in the near term. These are the company’s own figures; TechPulseMind has not independently verified them.
Why more HBM isn’t the whole answer
NVIDIA describes an AI factory as a memory hierarchy: GPU high-bandwidth memory at the top, where model weights and KV cache compete for capacity, then CPU memory, local SSDs and shared storage below, each trading speed for lower cost per bit.
The post uses a warehouse analogy. HBM is the warehouse and memory bandwidth is its loading docks — a bigger building holds more inventory, but the docks set how fast it moves. Once bandwidth is saturated, extra capacity can’t be put to work yet still adds cost and draws power. Engineers call that the memory wall.
Agentic workloads make the squeeze worse, NVIDIA says. KV cache holds the context a model needs for each active request, and agentic trajectories carry prior context across repeated model and tool calls, in some cases processing up to 3 million tokens. Retrieval-augmented generation adds more context on top.
The techniques NVIDIA says free up HBM
NVIDIA lists several methods, each aimed at a different source of waste:
- Smaller weights: the NVFP4 format cuts the space weights need compared with FP8, the lowest precision on Hopper. NVIDIA says recent models, including FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta Muse Spark, Step 5 Preview from StepFun and NVIDIA Nemotron 3.5 Super VL, shipped with NVFP4 checkpoints.
- Spreading the model: within an NVL72 system, weights can be distributed across up to 72 GPUs, leaving more HBM for KV cache and concurrent users.
- Offloading idle context: NVFP4 applied to KV cache shrinks its footprint, while NVIDIA Dynamo moves inactive context to CMX context memory storage and returns it before reuse. CMX partners listed include Cloudian, DDN, Dell Technologies, Everpure (previously Pure Storage), HPE, IBM, NetApp, VAST and WEKA.
- Keeping RAG indexes out of HBM: Nemotron Retriever models produce smaller embeddings, and the cuVS library builds and serves the index across CPU and storage, so only needed passages enter context.
NVIDIA also credits its Vera CPU: faster tool execution shortens the time KV cache sits idle, and its LPDDR memory, linked to Rubin GPUs over NVLink-C2C, gives paused context a nearby place to wait.
The numbers NVIDIA is putting on it
On the same AgentX workload at 100 tokens per second per user, NVIDIA says GB300 NVL72 delivers up to 10x higher throughput per dollar than the H200 platform, with HBM capacity per GPU roughly doubling. From Blackwell Ultra to Vera Rubin, HBM per GPU rose about 3% while agentic throughput per dollar rose up to 30x.
For operators, NVIDIA argues the metric that matters is how many useful tokens a factory delivers within its power budget while meeting response-time targets. The company points readers to a video explainer and technical deep dive for more detail.
Related reading
Sources
- More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge — NVIDIA Blog
Some links on this page may be affiliate links. If you buy through them we may earn a commission at no extra cost to you. See our affiliate disclosure.