LMCACHE

LMCache KV Cache Optimization Engine

Persistent Tiered KV Caching • TTFT Acceleration • Disaggregated Prefill/Decode

● ENGINE ACTIVE JSON Metrics API

⚡ Time-To-First-Token (TTFT)

18.4 ms -78% Latency
Accelerated prompt prefill via cached KV token blocks
Cache Hit Rate91.4%
Throughput Gain3.2x Tokens/sec
Non-Prefix ReuseEnabled

💾 Tiered Storage Hierarchy

4.8 GB Cached
Multi-tier distributed memory & object backend
L1: CPU / Memory RAM1.68 GB (Active)
L2: Redis Shared Tier2.16 GB (Warm)
L3: MinIO S3 Object Store0.96 GB (Cold)

🔄 PD Disaggregation & vLLM

4 Workers
Prefill to Decode KV transfer over high-speed mesh
vLLM Inference PodConnected
LiteLLM Proxy HookActive
Transport ProtocolLMCache IPC / gRPC

📊 LMCache Live KV Stream & Eviction Logs

[06:55:10] INIT: LMCache Engine v0.3.0 initialized on Kubernetes pod lmcache-78dcf4-k910.
[06:55:11] TIER-1: L1 CPU Memory Buffer allocated (2048 MB).
[06:55:12] TIER-2: Connected to Redis cluster at redis:6379 for cross-pod KV sharing.
[06:55:13] TIER-3: MinIO S3 Lakehouse bucket 's3://lakehouse/kv-cache/' mounted for cold persistence.
[06:55:14] HIT: Reused KV cache block (hash: 0x9f4a12c) for multi-round roadside diagnostic prompt (Saved 380 tokens).