Giving vLLM its RAM cache back: a three-tier KV cache on the Intel Arc Pro B70

From 479 preemptions to serving 128k contexts out of a 64 GB disk cache — including the upstream bug I had to patch to get there.

This is the sequel to Tuning Qwen3.8-27B on an Intel Arc Pro B70, where I squeezed llama.cpp up to 33 t/s at 128k context on a single Arc Pro B70. That setup had one feature I leaned on heavily and never fully appreciated: llama.cpp happily kept KV cache in system RAM and moved it to the GPU when needed.

Then I moved the stack to vLLM (nightly XPU build, OpenAI-compatible server, proper continuous batching) — and immediately traded that away without realizing it. vLLM's KV cache lives entirely in VRAM. On a 32 GB card running a 27B model at 128k context, that means the cache holds about one full-length request. The result was a pile of preemptions and 22-second time-to-first-tokens.

This post is about building vLLM's equivalent of what I'd lost — GPU → pinned RAM → NVMe tiering — and the upstream bug in the way that took three stages, one crash-loop, and porting an unmerged fix to get working.

The setup

Same hardware as last time: Intel Arc Pro B70 (32 GB), Ryzen 7 9700X, 32 GB RAM, 1 TB NVMe. The model is still Qwen3.8-27B, but now a GPTQ-Int4 checkpoint with the MTP draft layers preserved in BF16, served by vllm/vllm-openai-xpu (v0.27.2rc1.dev77 nightly) in Docker with ipc: host, fronted by the same LiteLLM proxy. fp8 KV cache, 128k context, MTP speculative decoding with 4 draft tokens, prefix caching on.

Why move off a tuned llama.cpp config that was hitting 33 t/s? The reasons that mattered for agentic coding: vLLM's chunked prefill keeps the GPU fed on very long prompts (the engine logs ~7.6k prompt tok/s during prefill windows), continuous batching behaves under concurrency instead of serializing, and the offloading machinery I'm about to describe exists at all. The trade: llama.cpp's flexibility — per-layer GPU offload, per-side KV quants — is gone. Whatever VRAM management you want, you do vLLM's way or not at all.

Reading the failure out of the logs

The startup log tells you everything if you know where to look:

Available KV cache memory: 5.64 GiB
GPU KV cache size: 140,530 tokens, Maximum concurrency for 131,072 tokens per request: 1.07x

That second line is the whole story. vLLM sizes a KV pool, then tells you how many concurrent full-length requests it supports. 1.07x means a single 128k request consumes essentially the entire cache. The second request doesn't wait politely — once the pool fills mid-generation, the scheduler preempts a running sequence, drops its KV (or swaps it), and recomputes later.

The live metrics made it visceral:

• vllm:num_preemptions_total = 479 and climbing
• Requests queued with num_requests_waiting_by_reason="capacity" — waiting specifically because there was no KV space
• Average queue time 12.3 s, average TTFT 22.4 s across ~800 requests
• The classic pattern in the periodic logs: Running: 1 reqs, Waiting: 1 reqs, GPU KV cache usage: 89%

One preempted 50k-token sequence costs a full re-prefill — about 7 seconds at this prefill speed — per preemption. A preempt storm is dozens of those.

Why the cache is so small: the model is hybrid

Qwen3.8 (the Qwen3.5 family architecture) is not a plain transformer. Of its 64 layers, only 16 are full attention; the other 48 are linear-attention (GDN) layers that carry a small fixed recurrent state (~2.25 MiB per sequence) instead of a growing cache. Only the 16 full-attention layers consume per-token KV: 16 layers × 2 (K and V) × 4 heads × 256 head_dim × 1 byte (fp8) ≈ 32 KB per token. A single 128k context is ~4.3 GB of KV. That's the arithmetic behind 1.07x — and no amount of KV quantization was going to fix a card where one context eats the whole pool.

(The good news: the same hybrid design is why the model runs at all on 32 GB. A dense 27B at 128k would need ~4× the KV budget.)

Side quest: the PCIe link that lies

Before trusting any offload design over PCIe, I measured the link. sysfs insisted the card was at 2.5 GT/s x1 — about 2 GB/s, a tenth of the link's real bandwidth. It kept saying that while transfers were running.

A direct benchmark said otherwise: 28.4 GB/s host→device, 24.7 GB/s device→host with pinned memory. That's real Gen4 x16 — and the number proves it: a Gen4 x8 or a Gen3 x16 link tops out around 16 GB/s per direction, so 28.4 can only mean x16 at Gen4 speed. The B70's sysfs link-speed readout is simply wrong (or reports an idle power state); the transfers are the ground truth. This matters because the entire offload design hangs on that bandwidth: 4 GB of KV crosses the real link in ~0.15 s, not the ~2 s the sysfs number would imply.

Stage 1: the boring config win

Three flags before anything clever:

--gpu-memory-utilization 0.94   # was 0.88
--max-num-seqs 4                # was 64
--max-num-batched-tokens 16384  # was 8192

gpu-memory-utilization is worth explaining, because it confused me: xpu-smi showed memory at ~100% and I assumed there was nothing to reclaim. But vLLM pre-allocates its KV pool at startup — the fraction you grant is carved out immediately, weights + activation workspace + KV. Raising 0.88 → 0.94 simply makes the pool bigger. Runtime utilization was always ~100%; that's the design, not a problem.

Result: KV pool 140,530 → 159,448 tokens (1.07x → 1.22x concurrency), and capping max-num-seqs at an honest 4 stopped requests from being admitted just to starve. Preemptions dropped from 479-and-climbing to zero or one under normal load. Decode speed unchanged.

That alone fixed the single-user experience. But 1.22x is still nothing for concurrency, and I wanted the llama.cpp behavior back.

Stage 2: the RAM tier, and the bug that came with it

vLLM's answer to llama.cpp's RAM cache is the OffloadingConnector: completed KV blocks are copied asynchronously (DMA over that Gen4 link) to pinned host memory, and later requests whose prefix matches get those blocks promoted back to the GPU. Crucially, this is a hierarchical prefix cache, not live-KV-in-RAM: your active working set still lives in the GPU pool; RAM extends what the engine remembers. For agentic coding — where every session re-sends a huge, mostly-identical codebase prefix — that's exactly the shape you want: the second turn of a session should not re-prefill 50k tokens.

Two config landmines for hybrid models, both documented here so you don't rediscover them:

1. block_size in the offload config asserts that all KV-cache groups share one block size. GDN + full-attention groups don't. Symptom: AssertionError in build_offloading_config during engine init, crash-looping the container. The fix is blocks_per_chunk: 1 (one GPU block per offload chunk), which skips that assertion entirely.
2. The tiering spec backs its shared CPU region with /dev/shm. With ipc: host, that's the host's tmpfs — Fedora's default is 50% of RAM = 16,352 MiB, and the spec wants 16,354. It missed by 2 MiB and refused to boot. mount -o remount,size=24G /dev/shm fixed it.

With those handled, the 16 GiB tier came up… and didn't work. Stores flowed — 8.9 GB pushed GPU→CPU at ~27 GB/s — but:

• kv_offload_cpu_cache_usage_perc stuck at 0.0
• External prefix cache hit rate stayed 0.0%
• Zero bytes ever moved CPU→GPU
• The acid test — re-sending a 40k-token document after deliberately evicting it from the GPU pool with 180k tokens of other traffic — paid the full 27 s re-prefill

Config was correct; transfers were happening; lookups never fired. I was deep into reading the scheduler's store-threshold code, assuming I'd misconfigured something, when the actual answer fell out of the issue tracker.

The bug was upstream, and exactly ours

vLLM issue #52735: "OffloadingConnector stores but never serves when MTP/EAGLE speculative decoding is enabled (hybrid GDN model, XPU)" — same GPU, same model family, same signature. (I found it the unglamorous way: a documentation-search site pointed me in the right direction, but the win came from searching the issue tracker with the exact symptom.)

The root cause is a lovely three-way interaction:

• Qwen3.8's MTP drafter isn't a separate model — its layer is merged into the target's full-attention KV group, so no group is annotated as a "drafter group"
• The scheduler's fallback, seeing spec decode enabled with no drafter annotated, decided every group was a drafter group
• Drafter groups get a "volatile tail" exclusion (the last chunk can still be rewritten by rejected draft tokens), and the volatile-tail pop then shrank every request's servable window to zero

So the connector stored everything and served nothing. The fix — PR #52771 — stops the all-groups fallback, lifts the volatile-tail exclusion once a request finishes, and widens the lookup query for all group types. It wasn't merged when I hit this, so I ported the three hunks into an idempotent patch script that runs at container start alongside the MTP/GDN patches the B70 already needed. (One risk to note honestly: the patch lets drafter KV be served one chunk stale. The target model verifies every draft token regardless, so worst case is slightly lower speculative acceptance — more on that below.)

Then the acid test again:

40k-token doc	before fix	after fix
cold (prefill + store)	27.6 s	28.5 s
repeat (GPU pool hit)	2.9 s	3.1 s
after 180k tokens of eviction pressure	27.0 s — full re-prefill	5.2 s — promoted from RAM

The metric that made it unambiguous: kv_offload_total_bytes_total{transfer_type="CPU_to_GPU"} went from literally zero, forever to 1.49 GB and climbing. The RAM tier was finally a cache instead of a write-only log.

Stage 3: the NVMe tier

With the connector actually serving, the disk tier is the same machinery plus a filesystem. One wrinkle: the volume group had zero free extents (single partition, fully allocated), so no LVM carve-out — instead a 64 GB loopback ext4 image, which gives the same kernel-enforced cap with none of the repartitioning:

sudo truncate -s 64G /var/lib/kv-cache.ext4 && sudo mkfs.ext4 -F /var/lib/kv-cache.ext4
echo '/var/lib/kv-cache.ext4 /mnt/kv-cache ext4 loop,noatime 0 2' | sudo tee -a /etc/fstab
sudo mount /mnt/kv-cache && sudo chown cchild:cchild /mnt/kv-cache

And the final engine config, via --kv-transfer-config:

{"kv_connector": "OffloadingConnector", "kv_role": "kv_both",
 "kv_connector_extra_config": {
   "spec_name": "TieringOffloadingSpec",
   "cpu_bytes_to_use": 17179869184,
   "blocks_per_chunk": 1,
   "eviction_policy": "lru",
   "secondary_tiers": [{"type": "fs", "root_dir": "/mnt/kv-cache",
                        "n_read_threads": 8, "n_write_threads": 8}]}}

One flag makes the disk cache survive restarts: PYTHONHASHSEED=0. Block hashes are chain-hashes seeded per-process; fix the seed and identical token content produces identical block filenames across restarts — which is what turns the NVMe tier from a scratch space into a persistent cache.

The validation sequence I cared about:

Test	Result
40k-token doc, cold (store)	27.7 s; 1.8 GB cascades GPU→RAM→NVMe while serving
repeat (GPU hit)	3.5 s
full container restart, same doc	4.2 s — served from NVMe

That last row is the money shot. Fresh container: GPU pool empty, RAM tier empty, the only copy of the document's KV is block files on disk. The request came back in 4.2 s, and the metrics show why: 1.49 GB promoted CPU→GPU (from disk through RAM), and GPU_to_CPU compute at 0.0 — the tokens were never re-prefilled. For reference, that same cold-start was a 22+ second TTFT in the pre-tiering regime, back when the queue was full of preemption victims.

The concurrency torture test

The whole point of the tiers is concurrency, so: N simultaneous ~127k-token contexts (my agent-workload stand-in), against a GPU pool that fits 1.22 of them.

Two concurrent requests — 89k and 127k prompts, 216k tokens combined: both completed, 1 preemption total.

Three concurrent requests — 89k + 127k + 122k, 338k prompt tokens against a 159k-token GPU pool: all completed in 83 s / 367 s / 216 s respectively, zero new preemptions. The scheduler interleaved them with chunked prefill (one running, two waiting, rotating), and aggregate prefill throughput hit ~9.2k tok/s — better than the 7.6k single-stream rate, because concurrent prefills keep the GPU fed. For scale: before any of this work, a single 40k request was generating preemption storms on its own.

The spill logic did exactly what the tiers promise: each 127k context is ~4 GB of full-attention KV, and GPU_to_CPU stores accumulated to 15.8 GB across the runs — the overflow drained to RAM while active sequences kept their working set on the GPU.

And the MTP trade-off? Mean acceptance length during the heavy concurrent runs was 3.0–3.5, against a ~2.4 baseline from before the patch. The "stale draft chunk" risk hasn't shown up as an acceptance regression in practice — if anything these runs were more repetitive (my synthetic docs), so treat it as "no harm observed," not "proven safe."

What it costs

• 16 GiB pinned RAM on a 32 GB box. Idle headroom after engine + tiers: ~2 GiB. It's fine for the LLM workload, but there's no room for a second memory-hungry service, and I watch dmesg for OOM-killer visits. The tier size is one env var if I need to shrink it.
• A patched nightly. The #52771 fix is upstream-open, so I carry a ported patch. It's idempotent and fails loudly if the nightly's code shape changes, but it's one more thing that isn't stock.
• The GPU pool is still 1.22x. Tiers extend the cache, not the working set — a cold 4-way distinct-128k storm would still queue. The fix for that is more VRAM, not more tiers.
• Quirks with edges. /dev/shm sizing, the 2 MiB failure, no built-in FS-tier quota (the loopback is the cap), stale per-config digest directories to prune if config changes accumulate.

The final config

Tier	Device	Size	Contents	Rehydrate 40k ctx
1	GPU pool	6.4 GiB	~159k tokens	instant
2	Pinned RAM	16 GiB	~500k tokens	~2 s
3	NVMe (loopback)	64 GiB	~2M tokens (~15 × 128k contexts)	~2.5 s transfer + gen

Engine flags that matter: --gpu-memory-utilization 0.94, --max-num-seqs 4, --max-num-batched-tokens 16384, --kv-cache-dtype fp8, MTP×4, and the --kv-transfer-config above. PYTHONHASHSEED=0 in the container environment. The full operator runbook — provisioning, validation gates, rollback per stage — lives in the repo next to the patch.

Takeaways for hybrid-GDN models on small GPUs

1. Read the Maximum concurrency startup line. It converts your VRAM budget into the only number that matters for long-context serving, and it's brutal on cards like the B70: one 128k context ≈ 4 GB of KV even at fp8.
2. gpu-memory-utilization is a startup reservation, not a runtime dial. If xpu-smi shows 100%, that's vLLM working as designed — raise the fraction to grow the pool.
3. Preemption count is the health metric. Not TTFT, not throughput — preemptions are where long-context engines go to die, and num_requests_waiting_by_reason="capacity" tells you it's KV-bound before users complain.
4. Prefix cache ≠ working set. Offload tiers fix repeat-context latency (huge for agentic coding) and absorb spill, but concurrency of distinct long contexts is still bounded by the GPU pool.
5. Speculative decoding + offloading is bleeding-edge. If your offload tier stores but never serves and you run MTP/EAGLE on a hybrid model, check #52735 before doubting your config.
6. Measure PCIe yourself. On this card, sysfs link reporting can't be trusted; a 30-second pinned-memory benchmark beats an hour of believing 2.5 GT/s x1.

Open questions

• Real-workload durability: my tests use synthetic documents. The next milestone is watching CPU_to_GPU promotions and disk-tier hits over weeks of actual agent sessions, and whether the 64 GB tier ever gets full enough for LRU eviction of disk blocks to matter.
• 128k-on-disk latency: the cold-start test promoted a 40k-token context (~1.3 GB of KV) in ~2.5 s. A full 128k context (~4 GB through NVMe→RAM→GPU) should scale linearly to ~8 s of transfer — still well ahead of a re-prefill, which costs 17 s even at the engine's best-window prefill rate (~7.6k tok/s) and closer to 80 s at the effective end-to-end rate our 40k test measured. But it's untested at that size.
• The patch's retirement: when #52771 merges, the patch becomes a no-op and comes out. Until then it's load-bearing.
• Reasoning effort: still medium globally, per the last post's caveat — the per-request control question is still open, and now it interacts with cache reuse: different efforts mean different token streams, which fragment the prefix cache.