Nvidia Claims 30x Agent Throughput per Dollar with Only 3% More HBM
Nvidia published a technical explainer arguing that a full-stack co-design approach can ease the HBM crunch and memory wall. Its stack combines NVFP4 low-precision compression, weight sharding across up to 72 GPUs in NVL72, LPDDR buffering on Vera CPUs, Dynamo context migration and CMX context storage to move inactive context off HBM. Citing the SemiAnalysis AgentX benchmark, Nvidia claims Vera Rubin NVL72 delivers up to 30x agent throughput per dollar versus GB300 NVL72 while HBM capacity per GPU grows only about 3%, signaling a shift from raw memory capacity to effective tokens per byte and per watt.
Full Report
IT之家 November 6 news, NVIDIA yesterday (November 5) released a technical interpretation, pointing out that with the rapid growth of large model parameter scale, context length, and agent applications, GPU memory is facing unprecedented pressure, HBM supply is tightening and prices are rising, and this will continue for the foreseeable future.
To break the current AI memory bottleneck, NVIDIA is exploring the introduction of a "full-stack co-design" philosophy, designing GPU, CPU, network, storage, and software as a whole in a coordinated way, so that every byte of memory generates higher value at every layer of the AI factory. IT之家 attaches the relevant images below:
During inference, expensive HBM must carry both model weights and store the KV cache (used to record the context the model needs to understand the current request). Agent workflows continuously carry historical context across multiple model calls and tool calls, and some scenarios need to handle as many as 3 million tokens; retrieval-augmented generation (RAG) also adds enterprise materials to prompts, further increasing memory usage.
NVIDIA compares HBM to a warehouse and memory bandwidth to loading docks: no matter how large the capacity is, if the speed of data moving in and out cannot keep up, a "memory wall" will form, causing added capacity to fail to effectively translate into service capability. Therefore, its solution emphasizes compressing data, placing it at the appropriate tier, and sending it back to the GPU in advance before it is needed.
In concrete operations, the NVFP4 low-precision format can compress model weights and KV cache; in the NVL72 system, model weights can be distributed across up to 72 GPUs; the large-capacity LPDDR memory of the Vera CPU can provide nearby buffering for paused contexts; and the Dynamo inference software migrates inactive contexts to CMX context storage and retrieves them before reuse. The RAG index can also remain on the CPU and storage side, avoiding long-term occupation of HBM.
Citing the SemiAnalysis AgentX benchmark, NVIDIA said that at 125 tokens per second per user, Vera Rubin NVL72 delivers up to 30 times higher agent throughput per dollar compared with GB300 NVL72, while HBM capacity per GPU increases by only about 3%.
At 100 tokens per second per user, GB300 NVL72 delivers up to 10 times higher throughput per dollar than the H200 platform. This shows that the key to AI computing competition is shifting from "how much memory there is" to "how many effective tokens can be produced per byte and per watt."