SC26: New Research Challenges GPU Utilization as Measure of Inference Infrastructure
Introducing new Inference Economics Scorecard to measure inference infrastructure by the useful work it delivers, as applied in an eight-hour benchmark with Nebius on Nvidia HGX B300 GPUs and Weka NeuralMesh
This is a Press Release edited by StorageNewsletter.com on November 17, 2026 at 2:01 pmWeka, an AI data and memory infrastructure company, in collaboration with market intelligence firm NAND Research, introduced the Inference Economics Scorecard, a new common framework for measuring inference infrastructure productivity and economic value by the useful work it delivers rather than by how busy it keeps GPUs.
The scorecard defines nine measures, including tokens per watt, cost per inference, and context retention. It sets four testing rules, including runs of at least eight hours and counting only successful output. Already applied in a benchmark with Nebius, the AI cloud company, it comes as inference is claiming a growing share of AI infrastructure spending and buyers hit power, memory, and rack-space limits.
Buyers Are Already Treating Inference Differently, New Research Finds
New research from 451 Research by S&P Global shows organizations are already adapting, projecting that inference will grow from 34% of AI infrastructure spending in 2026 to 46% by 2030 and finding that 41% of organizations running inference buy fundamentally different infrastructure for it than for the rest of their IT stack.
Why GPU Utilization Falls Short for Inference
For years, training was the dominant AI workload. Because training keeps GPUs busy with large, predictable jobs, GPU utilization became the default measure of AI infrastructure productivity. Inference, especially agentic inference, places very different demands on GPUs. An agent does not simply answer a prompt; it completes a task over many steps, and at each step it re-reads the context it built earlier. After the GPU evicts that context from memory to make room for other users and sessions, it must recompute it when the task returns, and the task makes no further progress until that work is done. The result is a GPU that is fully utilized but unproductive because it wastes cycles recomputing the same tokens multiple times.
The Inference Economics Scorecard measures inference systems at production scale: completed requests across many concurrent users, sustained over hours rather than minutes.
“The industry has been grading inference with a training-era report card. GPU utilization tells you the chip was busy. It cannot tell you whether the work was finished, repeated, or thrown away, and with agents, a lot of it is repeated,” said Liran Zvibel, co-founder and CEO, Weka. “Power, supply, and rack space are constraints now as much as capital. The new math required for inference infrastructure is useful work per watt, per rack, per $, at the response times your users expect. The Inference Economics Scorecard puts that math on paper, and Weka will be measured by it like everyone else.”
From Busy to Productive: How the Scorecard Works
The Inference Economics Scorecard defines nine measures of inference performance and economics: tokens per watt, tokens per rack, cost per inference, GPU goodput, time to first token and end-to-end latency, memory efficiency, operational density, agent concurrency, and context retention. It also sets the discipline for testing vs. them:
- The same workload and service targets across every system tested
- Measurement of the whole serving stack
- Runs of at least eight hours
- Only successful output counted
“The most expensive mistake in AI infrastructure right now is buying for peak throughput and paying for repeated work,” said Steve McDowell, principal analyst and founder, NAND Research. “Agentic workloads make that mistake easy to miss, because every step of a long task re-reads context that a busy GPU may have to rebuild from scratch. We designed the scorecard so buyers can hold any system to the same standard: finished work, measured over hours, with every failure counted. The Nebius benchmark shows what that discipline reveals, because it ran for the full eight hours and counted every request that failed or timed out.”
Putting It to the Test: Real Agent Load, Sustained for Eight Hours
Weka and Nebius recently applied the scorecard’s rules in a joint benchmark on Nvidia HGX B300 systems and reported results on several measures. The test ran on one eight-GPU node serving DeepSeek-V4-Pro to 512 active sessions drawn from a corpus of more than 5,000 real agentic coding sessions over eight hours per configuration, counting every failed and timed-out request.
Using Weka NeuralMesh with Augmented Memory Grid, which pools NVMe across the cluster as a shared context tier, the node produced 3.7x the output tokens per GPU-hour compared with HBM-only serving on the same GPUs, and 97% of successfully completed requests reached a first token within fives, up from 44%. On HBM-only serving, 60% of prefill work was recomputed from context the cluster had already processed earlier in the session; with Augmented Memory Grid, 6.8%. At the measured rate, the HBM-only node would need about 30 hours to produce what Augmented Memory Grid produced in eight.
“Peak throughput shows what a system can do for a few minutes, but our customers run agents for hours, days, or longer, so what matters is sustained useful work,” said Narek Tatevosyan, Vice president of Product, Nebius. “Over a full shift, the HBM-only configuration steadily lost ground as context fell out of memory and timeouts climbed, while Weka NeuralMesh kept first-token latency flat the entire run. That changes how we plan capacity: we size for finished work over a working day, not a benchmark peak, and the Inference Economics Scorecard gives us a common way to show customers that number.”
“Agentic inference is the workload that defines the AI factory, and it puts extraordinary demand on memory and context. The question at scale is no longer how much context fits on the GPU, but how far it can be extended across the cluster without being recomputed,” said Jason Hardy, VP, storage technology, Nvidia. “The Weka and Nebius results on Nvidia HGX B300 systems show what a shared context tier can do for sustained throughput and responsiveness, and as inference grows, buyers will need a clearer way to compare systems on the work they actually finish.”
Reproducible by Design: Open Traces and Tooling
The traces and tooling behind the benchmark are open-sourced so anyone can run the same workload on their own systems and compare results on equal terms. Because the traces come from real agentic coding sessions rather than synthetic load, and include the varied timing and complexity of real work, they give any team a realistic stand-in for production.













