In the tuning post I ended with numbers: 33 t/s, 128k context, done. Then one evening the agents just... crawled. First tokens took minutes. Instead of re-tuning anything, I looked at a dashboard — and it told me in about a minute exactly what was wrong. This post is about that dashboard: how it's built, and how I read each row when things get slow.
The setup: two boxes
The monitoring lives on my Unraid box (Docker images prom/prometheus:latest — currently 3.14 — and grafana/grafana:latest — 13.2). The reason it's not on the LLM box: Unraid is the machine that is always on and never gets re-imaged, while the LLM host gets rebooted and rebuilt when I break it. The monitoring should outlive the thing it monitors.
The LLM host (a Fedora box with the Intel Arc Pro B70) runs four containers that expose metrics endpoints:
| Container | Port | What it reports |
|---|---|---|
| vLLM | 8081 | The engine: tokens, TTFT, ITL, KV cache, preemptions, prefix cache, MTP |
| LiteLLM proxy | 4000 | The front door: in-flight requests, queue time, success/failure per model |
| node_exporter | 9100 | The host: CPU, RAM, disk, network |
| Intel GPU exporter | 9835 | The GPU: power, VRAM, temperatures (via the xe hwmon sensors) |
Prometheus scrapes the three interesting ones every 5 seconds (the node exporter every 15) and Grafana has one Prometheus datasource. The dashboard is 31 panels in 7 rows, a 3-hour default range, refreshing every 5 seconds.
The model side is the one from the tuning post, except the engine is now vLLM instead of llama.cpp — Qwen 3.8 27B with int4 weights, FP8 KV cache, 128k max context, prefix caching on, and 4-token multi-token-prediction (MTP) speculative decoding. vLLM gave me much better throughput in production; that story is for another post. Here the model is just the patient and the dashboard is the stethoscope.
vLLM hands you the dashboard parts for free
The important discovery: vLLM exposes a proper Prometheus /metrics endpoint as a first-class feature. You don't scrape anything custom — you point Prometheus at port 8081 and you get:
- Token counters —
prompt_tokens_totalandvllm:generation_tokens_total(prompt = prefill work, generation = decode work) - Latency histograms —
time_to_first_token_secondsandinter_token_latency_seconds, with real buckets you can take percentiles of - Engine state —
num_requests_running,num_requests_waiting,kv_cache_usage_perc,num_preemptions_total - Prefix caching — hits vs queries, and how many prompt tokens were served straight from cache
- Speculative decoding — draft tokens, accepted tokens, and (in recent builds) accepted tokens per draft position
There's also a nice trick where vLLM publishes its own config as a metric: vllm:cache_config_info is a gauge with labels like gpu_memory_utilization="0.94", cache_dtype="fp8", enable_prefix_caching="True", block_size="1664", num_gpu_blocks="118", kv_cache_size_tokens="159448". I learned what my engine was actually configured to do by reading a metric, which is how a monitoring stack should be.
Two caveats about those labels. Qwen 3.8 is a hybrid attention/SSM model (Qwen3-Next style), so the block math doesn't divide out the way a pure transformer's would — 118 blocks equals 159,448 cache tokens here, not some round multiple. Read kv_cache_size_tokens and ignore the rest. And that 159k-token budget is the number that matters for everything in this post: it's how much conversation the engine can hold before it starts throwing people out.
The dashboard
Four stat panels across the top give the live pulse, all using 15-second windows so they jitter — which is the point, they're a heartbeat, not a trend:
- Output tok/s (now) — decode throughput,
irate(vllm:generation_tokens_total[15s]) - Prompt tok/s (now) — prefill throughput
- TTFT p50 (now) — how long the first token takes
- ITL p50 (now) — how long each subsequent token takes
Below that, one row per subsystem. Here's what each row is for:
| Row | Panels | The question it answers |
|---|---|---|
| Throughput | output t/s, prompt t/s (with prefix-cache hits/s overlaid) | Is the engine busy, and is it spending its time on prefill or decode? |
| Speculative decoding (MTP) | acceptance rate %, mean accepted tokens per draft | Is the draft model earning its compute? |
| Latency | TTFT p50/p90, ITL p50/p99 | Which end is slow — the start or the stream? |
| Engine state | running / waiting requests, KV cache gauge, preemptions/s | Is it queuing, or is it thrashing? |
| Proxy (LiteLLM) | success vs failure rate, in-flight requests, queue time p50/p90 | Is the delay at the door or inside the engine? |
| GPU (Arc Pro B70) | utilization, VRAM %, board power (with the power cap line), hottest sensor | Is the hardware working, capped, or cooking? |
| Host | CPU %, RAM used, root disk %, network | Is the box around the GPU the problem? |
A few query details worth stealing, since each one exists because of a specific annoyance:
- The MTP acceptance panel divides two rates:
100 * sum(rate(accepted[1m])) / clamp_min(sum(rate(drafted[1m])), 1e-9). Theclamp_minis the trick — without it, an idle engine divides by zero and renders a giant spike instead of zero. - "Mean accepted tokens per draft" is
accepted / drafts. It's the single number that characterizes whether MTP is doing anything: with 4 draft positions, a healthy value sits around 2, and if it drifts toward 1 you're paying draft cost for nothing. - TTFT/ITL panels use
histogram_quantile(p, sum by (le) (rate(..._bucket[5m])))— thesum by (le)matters because vLLM labels its histograms per-engine and you want one distribution, not one per engine. - The prefix-cache panel overlays hits/s on the prompt t/s line, so a prefill that's supposed to be fast shows up as a thin line.
Reading it when something's slow
The whole point of the dashboard is that "slow" is not one thing. The table I run through in my head:
| It feels like... | Look at... | That means... |
|---|---|---|
| First token takes forever | TTFT p50 + prefix hit rate | Hit rate collapsed → the cache got evicted, every request is a cold prefill |
| First token takes forever | Running / waiting | Waiting is climbing → you're queued behind other requests |
| Tokens crawl | ITL + output t/s + MTP acceptance | Acceptance dropped → the draft model is losing on the new workload shape |
| Everything looks fine | KV gauge + preemptions | Preemptions climbing → the engine is evicting requests and making them re-run |
| The GPU is "idle" | Power, not utilization | On this card the utilization sensor under-samples; 326W at a 330W cap is not idle |
| Nothing is moving | The up targets, or all-flat panels |
It isn't slow, it's dead — the fix is a restart, not a re-tune |
That last row is the first check I make now, because I nearly did a re-tuning session on a dead engine.
The evening the dashboard earned its keep
September 1, 20:00–21:30. I had roughly nine agent sessions going at once — and my typical request is a ~74,000-token prompt (agentic contexts don't stay small), against a 159k-token KV budget. Here's what the dashboard showed, at 20:26 and 21:16:
| 20:26 | 21:16 | |
|---|---|---|
| TTFT p50 | 123 s | 400 s |
| Proxy in-flight / engine running / waiting | 9 / 2 / 6 | 5 / 1 / 3 |
| Prefix cache hit rate | 11% | 2.6% |
| Preemptions (cumulative) | 275 | 446 |
| Output t/s | 21 | 5.3 |
| GPU power (cap 330 W) | 326 W | 326 W |
| GPU utilization sensor | 25% | 25% |
The story it tells: the KV cache filled up, so vLLM started preempting running requests to make room for new ones. A preempted request doesn't pause — it gets its KV blocks back later and has to re-prefill from scratch. And the prefix cache — the thing that normally serves ~78% of my prompt tokens for free — got evicted in the churn, so every re-prefill was a full cold prefill of a 74k-token prompt. That's where "400 seconds to first token" comes from. Six hundred seconds at p90.
The dashboard's value was the diagnosis, not the number. Without it, "the model is glacial this evening" points at a hundred suspects — the quant, the MTP settings, the context size, the GPU. With it, the cause read off the panels in under a minute: the GPU was pegged at its power cap (the work was happening) while the useful-work rate collapsed and preemptions climbed. That's not slow compute. That's scheduling thrash — the engine spending its time re-doing work it had already done. The fix that night was "run fewer sessions at once"; the fix going forward is the alerts I haven't written yet (below).
Yesterday (Sep 2) was the other lesson. The engine went down three times — 18:33, 18:58, and 22:23, each outage 5–15 minutes. In the dashboard this looks nothing like the thrash: the stat panels go blank, the time series flatline, and the up{job="vllm"} target flips to zero. No slow panel to read, no histogram to quantify. Just: it's not slow, it's off. I restarted each time; the actual crash cause is in the container logs and is still an open thread. But the diagnosis — dead vs slow — took ten seconds, and that's a diagnosis you can't get from "the agent is taking a while."
The contrast across the two days is the whole argument for the stack: same "the model feels slow" feeling, two completely different causes (thrash vs death), and the dashboard tells them apart instantly.
The GPU row is a trap
One panel in that dashboard actively lies, and I'm keeping it anyway because the contrast is the lesson. The utilization sensor (read from the xe hwmon interface) sat at 0–25% for the entire day — including the 20:26 window where the card was pulling 326W against a 330W cap, 78°C at its hottest, VRAM at 99.5%. The sensor is under-sampled; it updates so sparsely that it mostly reports the GPU's idle state, even while the GPU is fully busy.
So the row is ordered by honesty: read power first. 47W means idle, 326W means "definitely working, and capped." VRAM is the second honest gauge — and note it reads 99.5% even at idle, because vLLM pre-allocates its whole KV budget at startup (gpu_memory_utilization 0.94). VRAM at 99% is the resting state, not a warning. KV cache usage (the vLLM-side gauge) is what warns you.
The Prometheus side
The whole config is small:
global:
scrape_interval: 15s
scrape_configs:
- job_name: prometheus
static_configs: [{targets: [localhost:9090]}]
# The LLM host. 5s for the jobs the dashboard diagnoses with in real time.
- job_name: vllm
scrape_interval: 5s
static_configs: [{targets: [10.0.0.18:8081]}]
- job_name: litellm
scrape_interval: 5s
authorization:
type: Bearer
credentials: <litellm master key> # :4000/metrics is auth-protected
static_configs: [{targets: [10.0.0.18:4000]}]
- job_name: gpu
scrape_interval: 5s
static_configs: [{targets: [10.0.0.18:9835]}]
- job_name: node
static_configs: [{targets: [10.0.0.18:9100]}]
Three notes. Why 5s on the interesting jobs: the stat panels run 15-second windows, and the engine-state gauges (running/waiting) are only meaningful if the scrape is frequent — at the 15s default, a request that ran for 40 seconds shows up as a 2-second blip. Why Bearer on LiteLLM: the proxy authenticates everything, metrics endpoint included; the scrape just carries the master key. Why :latest for both containers: it's a home stack on a box that's always up; I'd pin versions if I were alerting on this for a living. The trade is real — a Grafana minor upgrade that changes a panel schema is one docker compose pull && up -d away from breaking the dashboard — but I'm accepting it.
Reproducing it
On Unraid, two services:
services:
prometheus:
image: prom/prometheus:latest
ports: ["9090:9090"]
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- prom-data:/prometheus
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.retention.time=15d
grafana:
image: grafana/grafana:latest
ports: ["3000:3000"]
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=<pick something>
volumes:
- grafana-data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
volumes:
prom-data:
grafana-data:
The prometheus.yml is the one above. The Grafana side provisions its single datasource from a file (grafana/provisioning/datasources/prometheus.yml):
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
The dashboard itself isn't something you rebuild by hand — Grafana can export any dashboard as JSON over its API (GET /api/dashboards/uid/<uid>), and the UI's New → Import screen takes pasted JSON. So the whole 31-panel dashboard is a copy-paste artifact: pull the JSON from one box, paste it into the other, fix the target IPs, done. That's how I'd hand it to anyone else, and it's how I'd restore it after a bad Grafana upgrade.
What it doesn't do yet
Honest gaps:
- No alerts. The Sep 1 thrash was caught because I happened to be looking when it felt slow. With alerts, the 20:26 cascade would have pinged me 45 minutes earlier. The candidates are obvious from the post-mortem table:
up{job="vllm"} == 0for 2 minutes (fired three times yesterday), preemptions incrementing at all, KV usage over 90% for 5 minutes, and MTP acceptance rate dropping below the norm (50–70% is normal here — a sustained drop is the workload-shape canary). - Retention is 15 days. The incident data above is already aging out of the window. Good enough for "why was it slow last Tuesday," not for "is it getting slower month over month."
- The utilization panel is decorative. I'm keeping it as a monument to the lesson, but the panels worth reading are power, VRAM, and temperature.
- The proxy row counts what I send it, not what it costs. LiteLLM tracks per-model spend; I route
glm-5.3-flashanddeepseek-v4-flashto a hosted Ollama endpoint alongside the localqwen3.8, so "requests per model" is a cost proxy I haven't charted yet.
The open questions from the tuning post still stand — long-generation throughput, reasoning quality at medium — but this post adds a new one: now that I can see why it's slow, what do I actually do differently when it tells me? The preemption cascade suggests the KV budget and my session habits don't fit; the repeated crashes suggest the int4-on-B70 engine has an edge I haven't found. Both are now questions I can answer instead of vibes, and that's the part of this I didn't expect to like.