Tuning Qwen3.8-27B on an Intel Arc Pro B70: A Systematic Sweep for Agentic Coding

From 26 t/s to 33 t/s with a 2x context window — what actually moved the needle, and the tradeoffs that didn't show up in the benchmark.

The hardware and software

A single Intel Arc Pro B70 with 32 GB VRAM. 16 CPU cores, 32 GB system RAM. Qwen3.8-27B runs as a GGUF through llama.cpp's server image (ghcr.io/ggml-org/llama.cpp:server-intel) in Docker Compose, fronted by a LiteLLM proxy that exposes an OpenAI-compatible endpoint on port 4000.

The workload is agentic coding — autonomous coding agents that send long prompts, generate long responses (full file refactors, multi-step reasoning, tool-call chains), and hammer the endpoint session after session. Throughput and TTFT matter, but so does something less obvious: how many speculative tokens the model accepts, because that compounds over long generations.

This started with a stuck model download (the mmproj file froze at 3% mid-fetch). A container restart resumed it. That side-tracker turned into a full tuning campaign across roughly 70 configurations.

The starting point

The config I inherited was reasonable on paper:

Qwen3.8-27B-UD-Q6_K_XL.gguf @ 64k context
KV cache: q8_0 / q8_0
mmproj on GPU
MTP_PREDICTIONS=4, MTP_P_MIN=0.5
reasoning_effort: (unset — model default)

It benchmarked at TTFT 754 ms, TPS 26.5, MTP accept 52.2%, using 94.9% of VRAM. The numbers were fine, but two things stood out: the MTP accept rate was middling, and the model was burning a lot of tokens before answering. That second observation is where the story really starts.

The reasoning effort trap

Qwen3.8 is a hybrid-thinking model. By default it runs at reasoning_effort: xhigh — extra high. This is the single most important thing to know about running it.

The community had already flagged the consequences. One report measured roughly 10x context consumption versus the previous model generation, almost entirely because xhigh makes the model over-think every request. It produces long internal reasoning traces even for straightforward prompts. That inflates TTFT (you wait for the reasoning before any visible output), wastes tokens, and fills context faster.

The fix is one flag:

--chat-template-kwargs '{"reasoning_effort":"medium"}'
--reasoning auto
--reasoning-preserve

The measured impact was the largest single change in the entire campaign:

Setting	TPS	MTP%	Notes
default (xhigh)	26.5	52.2	baseline
`medium`	33.1	65.7	+25% TPS, +13.5pts MTP
`low`	33.7	66.2	marginally faster again
`none`	—	—	requests failed — model rejects this value

medium and low both lifted throughput about 25% and MTP accept rate by 13 points. none broke the chat template entirely (every request returned an error) — the valid range for this build is xhigh/medium/low.

The caveat nobody mentions

This is a throughput and token-cost win, not a free lunch. Lower reasoning effort means the model spends less time thinking before it answers. For simple tasks that's pure upside — you get the same answer faster. For hard tasks (tricky debugging, subtle architecture decisions, multi-constraint refactors), medium can produce worse answers than xhigh because the model has less reasoning budget to work through the problem.

The honest framing: medium trades reasoning depth for speed and cost. That trade is good for most agentic coding turns, where the model is doing mechanical work (write a function, refactor a module, call a tool). It's bad when you genuinely need the model to sit with a hard problem. The right setup probably involves per-request control — medium by default, xhigh on demand — rather than a single global setting. The benchmark measures the easy case; it does not capture the quality regression on hard tasks.

I'm running medium because most of my workload is the easy case. If your work is mostly hard reasoning, think carefully before adopting this globally.

The measurement harness

Hand-tuning is slow and unreliable. You change a flag, restart, eyeball a single run, and you can't tell signal from noise because t/s swings 3-4 points run-to-run on its own.

I built a sweep harness instead. It's a bash script with a list of config variants. For each variant it writes the overrides to .env, recreates the Docker container (not restart — restart reuses the old environment and silently ignores your .env changes, which bit me early on), waits for the health endpoint to come up, runs a fixed 5-run benchmark plus a warmup, parses the summary, captures VRAM from the GPU's fdinfo, and appends a row to a CSV. At the end it prints a table ranked by TPS.

# One variant = one line: name|KEY=VAL;KEY=VAL;...
./sweep.sh results.csv

Five sweeps covered about 70 configurations. Each variant took roughly 70-90 seconds (container recreate + model load + 5 benchmark runs), so a full 18-variant sweep ran in 20-30 minutes unattended.

A few things I learned building it:

docker compose restart does not re-read env_file. You need docker compose up -d to recreate the container with the new environment. This cost me a confusing afternoon where "changed" settings silently didn't apply.
VRAM readout needs root inside the container. The host can't read /proc/<pid>/fdinfo for a root-owned container process. The harness copies a small probe script into the container and runs it via docker exec.
The B70 reports VRAM as drm-resident-vram0 in fdinfo, not the generic drm-resident-vram key. Took a while to find the right field.
Parse stdin once. My first benchmark parser piped stdin through four greps sequentially; the first grep consumed stdin and the rest got nothing. Capture to a variable, then grep the variable.

The VRAM budget: quantization versus context

The B70 has 32 GB. The model weights and the KV cache compete for it. Bigger context means a bigger KV cache. Higher quant means bigger weights. You can't just max both.

The model is Qwen3.8-27B. The quants I had cached locally:

Quant	Size
Q6_K_XL	~25.9 GB
Q5_K_XL	~20.2 GB
Q4_K_XL	~17.9 GB
mmproj (vision)	~931 MB

At 64k context with q8_0 KV, Q6 was already at 94.9% VRAM. Doubling context to 128k roughly doubles the KV cache, which would blow past the budget unless I freed space. The levers for freeing space: drop the KV quant, drop the vision encoder to CPU, or drop the model quant.

The results:

Quant	Context	KV	mmproj	VRAM	TPS
Q6	64k	q8_0/q8_0	GPU	94.9%	26.5
Q6	128k	q4_0/q4_0	off	89.0%	32.5
Q6	128k	q4_0/q4_0	CPU	89.0%	34.5
Q5	128k	q8_0/q8_0	GPU	87.8%	33.7
Q5	256k	q4_0/q4_0	off	92.7%	34.5
Q4	128k	q8_0/q8_0	GPU	82.0%	27.3

Two things jumped out. Q6 fits at 128k — you don't have to drop to Q5 to get the bigger context, provided you free VRAM with KV quant and mmproj placement. And Q5 reaches 256k comfortably, which is a quarter-million-token window.

The Q4 row is interesting: despite using the least VRAM (82%), it was the slowest of the 128k configs. Smaller weights should prefill faster, but at Q4 the accuracy loss apparently causes more rejections and corrections. Q4 is not a free speedup — there's a quality floor below which throughput drops.

Asymmetric KV cache: K and V are not equal

This is the most counterintuitive finding, and it's backed by data from the llama.cpp community (discussion #23470, which ran careful per-side quantization tests).

The KV cache has two halves: the K (key) cache and the V (value) cache. They respond to quantization very differently. The community measured it directly: applying q4_0 to K alone reproduced the full quality collapse, while applying q4_0 to V alone changed roughly 1 in 500 outputs. Their conclusion: K is the side that can't be cheapened — spend the bits on K.

So the right split is asymmetric: keep K at higher precision, cheapen V. This contradicts the naive "just drop both to q4_0" approach, which quantizes the sensitive side.

My sweep tested the combinations, all at Q6/128k with mmproj on CPU:

K / V	TPS	MTP%	VRAM%
q8_0 / q8_0	32.6	62.0	95.0
q8_0 / q5_1	34.0	66.4	93.6
q8_0 / q4_1	33.1	62.5	92.4
q8_0 / q4_0	33.3	62.6	92.0
q4_0 / q8_0	33.3	62.8	92.0
q4_0 / q4_0	33.6	63.4	89.0

A few things to note:

V=q5_1 was the sweet spot. It beat both the cheaper q4_1 and the full-precision q8_0 on TPS and MTP, for about 1% more VRAM than q4_1. q5_1 uses a floating-point scale (versus q4_0/q4_1's fixed block scale), giving better dynamic range at the same 4-bit width. The benchmark can't measure the quality difference, but the cost is negligible, so it's the safer pick.
The "wrong way" asymmetry (K=q4_0, V=q8_0) was nearly identical to the right way (K=q8_0, V=q4_0) on this short-prompt benchmark. The K-sensitivity the community found shows up in long-context accuracy, not in a 500-token MTP metric. I chose to trust the research rather than the benchmark here: K stays at q8_0.
There is no q6_1 KV cache type. The valid types in this llama.cpp build are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1. The only step above q8_0 is f16, which doubles the V size and won't fit at 128k. q8_0 is the practical ceiling for K; q5_1 is the sweet spot for V.
iq4_nl failed to load as a KV cache type on the Intel SYCL backend, despite being listed in --help. It's not actually implemented for KV cache here.

One thing worth noting: I also looked into TurboQuant (a 3-bit PolarQuant+QJL scheme from Zandieh et al., ICLR 2026) which compresses KV cache ~4.9x with near-zero accuracy loss. It exists, but it's implemented for the ik_llama.cpp fork, not stock llama.cpp. The server-intel image I'm using doesn't have it. If you ever want to push to 256k on Q6, compiling that fork is the path.

MTP: predictions and the acceptance threshold

Multi-token prediction (MTP) is Qwen3.8's speculative decoding. A small draft model predicts several tokens ahead, and the target model verifies them in one forward pass. Accepted tokens are nearly free; rejected tokens cost a wasted draft pass. Two knobs control it: MTP_PREDICTIONS (how many tokens the draft predicts per step) and MTP_P_MIN (the confidence threshold below which a draft token is rejected).

The intuition: more predictions gives the draft more chances to match, raising accept rate. But each prediction costs draft-model compute, and if p_min is too low you accept bad drafts that the target model then has to correct, which slows you down. The sweet spot is where accept rate is high enough that the draft work pays for itself.

I ran a 20-cell grid (MTP 2-6 × p_min 0.3-0.6) plus some draft-KV and batch variants:

MTP	p_min	TPS	MTP%	VRAM%
2	0.3	32.7	58.5	92.7
3	0.4	34.0	64.3	93.1
4	0.5	32.8	65.2	93.6
5	0.4	33.1	69.4	94.1
6	0.5	33.0	69.9	94.5
6	0.3	31.9	71.6	94.5

Clear patterns:

MTP=2 caps out around 58% accept. Too few predictions — the draft rarely gets a long enough match.
p_min=0.4 is the sweet spot across MTP 3/4/5. Lower (0.3) over-accepts bad drafts and TPS drops; higher (0.6) rejects too many and accept rate drops.
MTP=5/6 both fit at 94.1-94.5% VRAM and push accept rate to ~70%. The extra draft compute costs a little TPS (vs MTP=3) but the accept rate jump is large.
MTP=8 collapsed (tested in an earlier sweep): the draft model's KV cache and compute pushed VRAM over the edge and the target model spilled to CPU, dropping TPS to 2.6. There's a hard ceiling somewhere around 6-7 on this hardware.

The draft model's own KV cache quantization barely mattered — f16, q5_1, and q8_0 draft all gave nearly identical accept rates. The draft context is tiny, so its cache precision doesn't have room to affect anything. q8_0 draft is fine.

The vision encoder: a free trade

The mmproj (vision) encoder takes about 3.5 GB of VRAM when loaded on GPU. For agentic coding, vision is rarely used but nice to have available. Three options:

mmproj placement	VRAM	TPS	MTP%
on GPU	high	baseline	baseline
on CPU (`--no-mmproj-offload`)	-3.5 GB	slightly higher	slightly higher
off entirely (`--no-mmproj`)	-3.5 GB	neutral	slightly lower

Offloading to CPU freed ~3.5 GB of VRAM for the LLM and slightly improved TPS and MTP — the GPU is better spent on the main model. Disabling mmproj entirely didn't help further (and loses vision). So --no-mmproj-offload is the free win: keep vision available, run it on CPU, spend the GPU on the LLM.

Why MTP accept rate matters more than raw TPS for agentic coding

This is the part the benchmark undersells.

Raw decode TPS measures tokens per second after the first token. It's a clean number and easy to compare. But it only counts tokens the target model generated. It doesn't count tokens the target model didn't have to generate because the draft model already produced them correctly.

Each accepted draft token is one token the target model verified cheaply instead of generating from scratch. Higher accept rate means more tokens per expensive target forward pass. Over a short generation (500 tokens) the effect is small and dominated by warmup and graph-compile cost. Over a long generation — a full file refactor, a multi-step tool-call chain, a long reasoning trace — it compounds.

Concretely: at 64% accept (MTP=3), about 64% of tokens come from the draft. At 69% (MTP=5), about 69% do. Over a 2000-token generation, that's ~100 extra tokens the target model never produced. The benchmark shows MTP=3 as ~0.9 t/s faster in raw decode; the real coding session likely inverts that because the higher-accept config does less total target-model work.

This is also why p_min matters more than it looks. Dropping p_min from 0.5 to 0.4 raises accept rate several points with almost no TPS cost — the extra accepted tokens are mostly good. Dropping to 0.3 raises it further but starts accepting bad drafts that cost correction passes, and TPS falls. The threshold is a quality/speed tradeoff disguised as a single number.

The honest limitation

I'm asserting this based on reasoning about the mechanism, not on a long-generation benchmark. The 500-token sweep can't actually measure the compounding effect — it measures raw decode, which is exactly the metric that underrates high accept rate. To prove the claim properly I'd need to benchmark with max_tokens=2000+ and measure wall-clock per response, not per token. I haven't done that yet. The config is running and I'm observing it on real sessions, but I don't have a clean number for "effective throughput on a 2000-token agentic generation." Take the claim as informed-but-unverified until I run that test.

The final config

MODEL_FILE=Qwen3.8-27B-UD-Q6_K_XL.gguf
MODEL_CTX_SIZE=128000
KV_CACHE_TYPE_K=q8_0
KV_CACHE_TYPE_V=q5_1
DRAFT_KV_CACHE_TYPE_K=q8_0
DRAFT_KV_CACHE_TYPE_V=q8_0
MMPROJ=1
MMPROJ_OFFLOAD=0
SPEC_TYPE=draft-mtp
MTP_PREDICTIONS=5
MTP_P_MIN=0.4
REASONING_EFFORT=medium
BATCH_SIZE=4096
UBATCH_SIZE=2048

Result: TTFT ~700 ms, TPS ~33, MTP accept ~66-69%, VRAM ~94%.

Compared to the starting point (Q6@64k, q8_0/q8_0 KV, MTP=4/p_min=0.5, xhigh reasoning): 2x the context window, ~25% more throughput, ~14 points higher MTP accept rate.

The changes and what each contributed:

reasoning_effort=medium: the single biggest lever — +25% TPS, +13.5pts MTP. Tradeoff: less reasoning depth on hard tasks.
MTP_PREDICTIONS=5, p_min=0.4: +4.5pts MTP accept over MTP=4/p_min=0.5, ~no TPS cost. Better effective throughput on long generations (unverified by long-gen benchmark).
K=q8_0, V=q5_1: +1.4pts MTP and slightly higher TPS vs q8_0/q8_0, at lower VRAM (93.6% vs 95%). The asymmetric split is the research-backed call.
mmproj on CPU: freed 3.5 GB VRAM, enabling 128k context at Q6. Slight TPS/MTP improvement.
ctx 128k (from 64k): 2x context, no throughput cost once VRAM was freed.

Things that didn't help

Worth listing, because negative results are useful:

MTP=8: collapsed to 2.6 t/s. Draft model KV/compute pushed VRAM over the edge and the target model spilled to CPU. Hard ceiling around 6-7 on this hardware.
KV q4_0/q4_0: fastest raw decode but lower MTP and, per the research, worse long-context quality (K was cheapened). The symmetric cheap option isn't the right call.
Draft KV at f16: no measurable accept-rate improvement. Draft context is too small for precision to matter.
ubatch=512 (the "53 t/s winner" config from the community): slower on the B70. That config was tuned for an NVIDIA RTX 5060 Ti; the B70's SYCL backend prefers larger ubatch.
batch 8192 / ubatch 4096: peak TPS (34.3) but at 99% VRAM. Too tight for production — any real session OOMs or thrashes.
reasoning_effort=none: broke the chat template entirely.
iq4_nl as KV cache type: listed in --help but not actually implemented for KV on the Intel backend. Fails to load.
V=f16 or bf16: won't fit at 128k (doubles the V cache). Even before measuring, the math said no.

How to reproduce

The stack is Docker Compose with an env-driven entrypoint. I added the following knobs to llm-server-entrypoint.sh, all defaulting to current behavior so nothing breaks:

KV_CACHE_TYPE_K / KV_CACHE_TYPE_V — target model KV cache quantization
DRAFT_KV_CACHE_TYPE_K / DRAFT_KV_CACHE_TYPE_V — draft model KV cache quantization
DRAFT_N_GPU_LAYERS — draft model GPU offload layers
MMPROJ / MMPROJ_OFFLOAD — vision encoder on/off and CPU offload
REASONING_EFFORT — Qwen3.8 thinking depth (xhigh/medium/low)
MTP_PREDICTIONS / MTP_P_MIN — speculative decoding depth and acceptance threshold

The sweep harness (sweep.sh) iterates a list of config variants, recreates the container per variant, waits for the health endpoint, runs a fixed benchmark, captures VRAM, and records everything to a CSV. Add a variant as one line (name|KEY=VAL;KEY=VAL;...), run ./sweep.sh results.csv, come back to a ranked table.

All sweep results are saved in sweep-results.csv through sweep5-results.csv.

Open questions

Long-generation throughput: the claim that higher MTP accept rate beats higher raw TPS on real agentic sessions is mechanistically sound but not yet verified with a max_tokens=2000+ benchmark. That's the next test.
Reasoning quality at medium: I'm trading reasoning depth for speed. For my workload (mostly mechanical coding) that's the right trade. For heavy reasoning it may not be. The right long-term setup is per-request reasoning effort, not a global default.
K precision in long context: the research says K can't be cheapened and the short-prompt benchmark can't distinguish K=q4_0 from K=q8_0. A long-context accuracy test would settle whether K=q8_0 is actually worth the VRAM over K=q4_0.