Running Qwen 3.8 Flash Next on nextsycl: 63 GB of Model on a 32 GB Card

The model writing this sentence is the one being benchmarked — Qwen 3.8 Flash Next, 63.4 GiB of weights, on a single 32 GB Intel Arc Pro B70, through nextsycl. It reads about 1,200 tok/s, decodes about 78, and has just reused 99.9% of this conversation's prompt.

What nextsycl is

It's a Rust runtime for language, image, video and audio models on Intel Arc, with hand-written SYCL kernels. No CUDA, no PyTorch, no Vulkan, no Python, no Node. Every architecture gets its own engine tuned end to end, and each engine is a library you can link against without the server. Upstream describes itself as "a curated Ollama for everything a local machine needs", with a Docker-style CLI (nextsycl llm start | stop | ps | logs), model cards as YAML with pinned SHA-256s, and a single control plane — the front door on :8000.

The engine that matters here is qwen4exp, which serves the Qwen 3.8 Flash Next GGUF family: Gated DeltaNet + QSA attention, hyper-connections, a hashed per-layer embedding, and 512 experts per layer, with MTP speculative decode on by default. Greedy output with drafts equals output without — verify passes and batches bit-exact, which is the kind of guarantee you want before trusting a port for daily work.

It plugs into my stack the same way the other engines do: shared llm-net Docker network, alias llm-server, port 8081, LiteLLM on 4000 as the OpenAI front door.

Why it runs here when GRIMOIRE can't

I already had two ways to run this model, and one of them refuses to load on this box. The difference is entirely memory.

	GRIMOIRE (nvidia/…-NVFP4)	nextsycl (ISTA-DASLab/…-IQ2_XS)
Experts	68 GB (NVFP4)	~33 GiB (2-bit)
Experts in VRAM on one B70	~14 GB	~20–25 GiB
Experts in pinned host RAM	~49 GB	~9–13 GiB
PLE n-gram table	51 GB, file-backed	27 GiB, file-backed

GRIMOIRE's tiered loader keeps every expert in either VRAM or pinned host RAM — there is no SSD tier for experts, only for the PLE table — and it refuses the load before any GPU work if pinned RAM is short. On this host (30 GiB RAM, ~22 GiB available at the time) it wants ~49 GB pinned plus 8 GB headroom. No value of GRIMOIRE_EXPERT_VRAM_PER_LAYER rescues it; it's short by ~25 GB. That backend is correct and ready, it just lives on a bigger machine.

nextsycl's IQ2_XS experts are roughly half the size, and a single B70 holds about 24–25 GiB of the 33 GiB before the session's KV is carved out, leaving ~9 GiB for pinned host memory — inside this box's budget. The 27 GiB PLE table is read row-by-row from the file (16 rows per token), so it never has to be resident. It is still the tight path. The current live state, from nextsycl status:

local - up 31m, context 262144, MTP on, 50 request(s) served
GPU  CARD               VRAM USED       FREE   EXPERTS VRAM/HOST   POWER
0    Arc Pro B70    30.7GiB / 31.9GiB   1.2GiB   14978 / 9598      194W
prompt cache: 5 checkpoints, 3.5GiB of 4.0GiB, 39 evicted; the live session holds 45,883 tokens
  on disk: 85 checkpoints, 31.9GiB of 32.0GiB

14,978 of 24,576 experts in VRAM, 9,598 in pinned host RAM. The pinned tier needs ipc: host and ulimits.memlock: -1 in compose, because pinned USM is mlocked and can't be backed by a small container /dev/shm.

The prompt-cache bug was the real story

For weeks, prompt caching was doing nothing for me — 0% reuse on every agent turn. The cause was three small things stacked:

1. Checkpoint stops keyed on <|user|>, which is a GLM token. Qwen's ChatML uses <|im_start|>user, and <|user|> tokenizes into five unrelated pieces, so the only stop was the end of the prompt.
2. A prompt-end checkpoint can never be a prefix of the next turn, because the next turn re-renders the previous assistant turn without its old thinking. The token streams diverge immediately.
3. Live reuse required the whole live session to be a prefix, not the longest common prefix — so a genuinely shared prefix was thrown away.

Net effect: every agent turn re-read the entire transcript at ~800 tok/s. 6.6 s at 4.8K, 11 s at 8.6K, 24 s at 19K — before a single token was decoded. That's the worst case for agentic coding, because every turn resends the whole conversation.

I wrote a patch (stops() checkpoints every turn boundary), ran it as a local fork, and got ~99.8% reuse. Then upstream shipped the same fix officially on 2026-10-17 (commit 5fbbf32, "Prompt caching for every language model, as directive 1"), with user-turn markers moved into nextsycl_tok::USER_TURNS and a machine-checked rule in the test suite: templates must keep a cacheable prefix over three turns. That's the part I like — the correctness property is enforced by a test, not by a comment. My fork is gone.

What that looks like now, four growing turns over a 72.8 KB source file:

run	turn	prompt	reused	fed	read	TTFT
cold	1	20,051	0	20,051	22.8 s	23.4 s
cold	3	20,107	20,054	53	0.83 s	1.42 s
warm	2	20,079	20,054	25	0.34 s	0.95 s
warm	4	20,135	20,110	25	0.37 s	0.93 s

~99.9% reuse, TTFT 17 s → 0.9 s. The live server log makes the same point as it goes:

[request #47: 34042 prompt tokens (20352 reused from cache, 13690 fed in 22.0 s, 3 checkpoint(s) saved),
 199 generated in 3.7 s (54.21 tok/s), drafts 129/166 accepted, stop]

A tool call that adds 2k tokens now reads 2k tokens, not the whole 36k transcript — 7.7 s instead of the ~45 s a full re-read at that depth would cost.

The numbers

Benchy v1, one session, ctx 262144, MTP on, greedy, 256 new tokens:

Input tokens	PP (tok/s)	TTFT (s)	TG (tok/s)	Drafts accepted
30	78	0.40	72.9	79%
7,914	1,138	6.99	76.9	73%
39,783	1,215	32.8	74.1	70%
127,303	1,203	106.0	77.8	81%

Three things fall out of this. Decode is bound by expert traffic, not prompt length — it's flat from 2K to 128K. Context size barely matters (allocating 256K instead of 32K changes things by 1–2%). MTP is a single-stream win, not a throughput win: 44 tok/s without drafts, ~78 with them, but at 2+ concurrent sessions the draft/verify benefit is spent and aggregate falls back to the no-MTP number.

Concurrency doesn't raise throughput at all on this MoE — ~35–38 tok/s aggregate whether 1, 2 or 4 clients, with latency scaling linearly. Two 256-token requests finish faster sent serially (~8.6 s wall) than concurrently (14.7 s). So PARALLEL=1 with a single 256K session is the right setting for one person driving several agents: requests queue, and the session swaps context per turn, restoring the other user's checkpoint from the prompt cache. I tried PARALLEL=2 on 2026-10-17, measured it, and reverted it — it halves throughput, triples latency, and drops VRAM below the engine's 1.5 GiB guard.

The host RAM, not VRAM, is the ceiling. Every config fills the card to ~2 GiB free by pushing overflow experts to pinned RAM, and pinned memory is mlocked — it can't be reclaimed. The cliff is visible in the quant sweep: IQ3_XXS at 20.2 GiB pinned runs fine, IQ3_S at 27.5 GiB falls off to 0.1 tok/s, not because of the quant but because the host stack the GPU waits on (PLE preads, expert gathers) started thrashing.

Build and run

./build.sh          # oneAPI image + SYCL kernels + Rust binary (~30 GiB of output)
./pull-model.sh     # 63.4 GiB, two shards, SHA-256 verified, resumable
docker compose -f llm-backends/nextsycl/docker-compose.yml up -d

Two local deviations from upstream, both documented in my backend README: upstream's build.sh hardcodes podman and this host has docker, so the same steps run through NS_CONTAINER_ENGINE; and upstream's kernel build expects /opt/onednn with headers that this base image doesn't ship, so I build the llm kind only and point DNNL at oneAPI's own oneDNN. I only serve an LLM, so image/video/diffusion kernels are irrelevant. Cold load takes ~140 s (63.4 GiB off disk plus expert placement); warm, 50 s.

The 2026-10-21 rebuild (pinned to 3e3bf0f, tag v26.1020.008) is the one running now: prompt experts on int8 kernels, a prompt checkpointed where its reply begins too, and loads serialized by the GPU run lock / VRAM planning. Decode is unchanged; prompt processing is measurably faster (+28% at 2K, +8% at 8K, +4% at 40K).

Why I like this one

Three things that make it feel different from the other engines in the folder:

• Speed claims are measured, not asserted. Every model card has numbers at each context length, and the repo's own rule is that nothing is committed until it beats the base with the same results. Bit-exact verify against a reference run is a real standard.
• Correctness properties live in tests. "Templates must keep a cacheable prefix over three turns" is a cargo test, which is why the cache fix didn't rot.
• It respects the memory budget instead of lying about it. It fails loudly on an allocation it can't make rather than hanging the card, and it reads the PLE table from disk rather than pretending 27 GiB is resident.

The honest caveat: this is one 32 GB card doing a job it was never sized for. 1.2 GiB of VRAM free, 4.9 GiB of RAM available, and a 27 GiB table being paged from NVMe. It works because the accounting is explicit. If I had a second card, all the experts would sit in VRAM, the pinned tier would collapse to zero, and the whole thrash conversation would go away — which is exactly what upstream's two-card figures show.