Tensormesh Launches Tensormesh Platform to Cut LLM Inference Costs by Reusing KV Cache at Enterprise Scale

Tensormesh, creators of LMCache, an open-source KV caching technology, announced Tensormesh Platform, the data management layer for AI inference.This multi-tier, scalable KV cache management software deploys across infrastructure with an enterprise Kubernetes deployment. With intelligent, AI-native data at its core, Tensormesh Platform enables enterprises to optimize the efficiency, cost, and quality of every inference they run. This marks the arrival of the first enterprise-ready KV cache management offering that works with any LLM, inference orchestration software, and storage vendor.

“LLM workloads recompute the same context endlessly. That wastes GPU time and inflates time-to-first-token,” explained Junchen Jiang, CEO, Tensormesh. “Tensormesh Platform addresses these costly problems, bringing production-grade KV cache reuse to enterprise on-prem inference stacks and making it operational at fleet scale.”

Tensormesh Platform includes the management capabilities enterprises need to run KV cache in production:

• Security: protects cached data across every tier
• Observability: provides real-time visibility into cache performance
• Resilience: HA keeps inference workloads resilient through failures
• Policy Management: controls how cache data is stored, shared, and evicted
• Multi Tenancy: tenant management lets organizations serve multiple teams or customers from shared infrastructure, with each tenant’s data isolated and governed

“Tensormesh delivered strong results in our P2P KV cache transfer and resiliency benchmarks across multiple models. We’re excited to see them continue pushing the ecosystem forward,” said Patrick Lundquist, senior software engineer, Nebius.

Enterprises deploying Tensormesh Platform also benefit from:

• Tiered Caching: KV Cache is managed across HBM, CPU, and storage layers automatically
• Prefix Caching: reuses the computed KV cache for the opening part of a prompt
• NonPrefix Caching (CacheBlend): reuses computed blocks anywhere in the prompt, not just at the start, making Tensormesh Platform ideal for RAG
• Peer-to-peer KV transfer: fetches KV cache directly from any peer in the fleet
• Advanced cache controls: allows users to pin, look up, move, compress, and clear entries via controller APIs, per workload or fleet-wide

“Customers are under tremendous pressure both to drive efficiency through AI and to manage the costs of AI infrastructure that powers such capabilities,” said Rhys Oxenham, GM, AI, SUSE. “The combination of Tensormesh and our SUSE AI Factory built on our open infrastructure platform empowers organizations to do both.”

The Tensormesh Platform is available.