Everyone assumes serious local inference means an RTX 4090 or a rented A100. It doesn't. For about $500 you can put an Intel Arc B60 in a box and serve a 30B model at ~40 tokens per second — 24/7, nothing leaving the machine, $0 cloud bill. There's one catch that turned out to be the entire story: the 30B that flies is a mixture-of-experts model, and a dense model half its size crawls on the same card. Here's the production data, and why.
why Intel Arc / Battlemage at all
The case for the Intel Arc Pro B60 is boring and it's the whole point: it's about the cheapest way to get a 24GB-class card into a box for local LLM serving. Two honest caveats up front, because they matter later: the card is marketed as 24GB, but under Vulkan the usable pool is ≈21 GB — that's the number you actually have to fit a model into. And it runs on the open Xe (Mesa 3D) driver with llama.cpp's Vulkan backend, not CUDA.
For local inference, VRAM capacity is the gate — it decides which models and context lengths you can hold at all — and Intel is currently the cheapest path to this much of it that isn't a years-old datacenter card with no cooling story. The tradeoff is real: Arc's Vulkan backend trails NVIDIA's CUDA stack on raw kernel optimization, and, as you'll see, that gap decides which architectures are viable here. This isn't the card for a dense 70B at high concurrency. For a 30B assistant answering real traffic all day at $0 marginal cost, it's excellent — you just have to pick the right kind of 30B.
the stack
The serving path is llama.cpp built against Vulkan — no vendor SDK to babysit, because the Vulkan backend treats the Arc card as a generic compute device. The companion intel-arc-llm-stack repo wraps this into a one-command OpenAI-compatible server (with an IPEX-LLM SYCL path and model hot-swap on top), but the core is this. Building it is genuinely a one-liner once the Vulkan SDK is present:
build — llama.cpp, vulkan backend
# with the Vulkan SDK + Xe / Mesa drivers installed cmake -B build -DGGML_VULKAN=ON cmake --build build --config Release -j # confirm the B60 shows up as a Vulkan device (~21 GB usable) ./build/bin/llama-cli --list-devices
And serving the production model — the 30B MoE, quantized IQ4_XS — as an OpenAI-compatible endpoint:
serve — 30B-A3B MoE, IQ4_XS · one model at a time
./build/bin/llama-server \ -m ./models/Qwen3-Coder-30B-A3B-Instruct-IQ4_XS.gguf \ -ngl 999 # push all layers to the GPU \ -c 8192 # context window \ --host 0.0.0.0 --port 8080
// hard rule One model on the B60 at a time. Load a second while the first is resident and the driver spills to system RAM — from there it's a swap death spiral, not a graceful slowdown. Stop one before you start another.
the finding: MoE crushes dense on Arc
I tested two ~30B-class coder models at the same quant (IQ4_XS), same card, same llama.cpp Vulkan build, same prompt. One is dense; one is a mixture-of-experts with 3B active parameters per token (the "A3B"). The result wasn't close:
| Model | Arch | Size | Quant | Speed (warm) | Status |
|---|---|---|---|---|---|
| Qwen3-Coder-30B-A3B | MoE · 3B active | 22.8 GB | IQ4_XS | ~40 tok/s | ✅ production |
| Qwen3.8-27B | dense · 27B active | 14.4 GB | IQ4_XS | 4.4 tok/s | ❌ rejected |
// warm, single-request, measured on my rig. the MoE figure is a production average over 30+ days of 24/7 traffic; the dense figure is 3 measured runs (cold first run ~6.9 tok/s, then steady at 4.4). full methodology + raw numbers: intel-arc-llm-benchmarks.
Read that again: the 30B MoE is ~9× faster than the 27B dense model — while being the larger file on disk. The dense model isn't slow because it's too big to fit; at 14.4 GB it fits the ≈21 GB card with room to spare. It's slow because of what the card has to do per token, not what it has to hold. The MoE lives in production at a steady ~40 tok/s with 99%+ uptime over a month of real traffic. The dense model fell below my production floor by 5×, and I rejected it.
why this happens
The mechanism is the useful part, because it tells you how to choose models for this hardware. Intel Arc's Vulkan backend is compute-bound, not memory-bound. On NVIDIA, dense inference is usually gated by how fast you can stream weights out of VRAM; on Arc's Vulkan path, you hit the compute wall first. So the thing that dominates throughput here isn't the model's size — it's how many parameters actually get activated per token.
| Model | Active params / token | Compute load |
|---|---|---|
| Dense 27B | 27B (100%) | heavy |
| MoE 30B (A3B) | 3B (10%) | light |
// a MoE routes each token through a small slice of its experts. cut active params ~10× (27B → 3B) and, on a compute-bound backend, you get roughly ~10× the speed.
That 10× cut in active compute is where the ~9× speedup comes from. And it compounds with what Arc's Vulkan backend is missing versus CUDA — no speculative decoding, no low-bit (INT4/INT8) tensor kernels, no Flash-Attention equivalent. Those absences punish dense models hardest, because a dense model needs every one of those optimizations to stay competitive. A MoE gets its speed from architectural sparsity instead of kernel tricks, so it barely cares that the tricks aren't there. Until Intel ships those kernels on Vulkan, the conclusion holds: on this card, match the architecture to the backend — pick MoE.
the gotchas worth writing down
- 20 tok/s is the hard production floor. Below it, real users perceive the assistant as "slow," so it's a pass/fail gate, not a nice-to-have. The dense 27B's 4.4 tok/s isn't "a bit under" — it's disqualifying.
- IQ4_XS is the sweet spot for the A3B. At that quant the 30B MoE lands at ~40 tok/s with quality that holds up on real coding prompts, and the 22.8 GB file stays resident on the ≈21 GB card. It's the recipe I run in production.
- Never co-load two models. With one model already near the top of VRAM, a second one tips the driver into spilling to RAM, and it degrades hard from there. One model at a time, always.
- "24GB" is ≈21 GB in practice. Size your model against what Vulkan actually exposes, not the box-art number — a build that looks like it fits the marketing spec can miss the real pool.
- Cold vs warm is real but small here. The dense model's first run measured ~6.9 tok/s before settling to 4.4; production servers are always warm, so I benchmark warm and treat cold-start as a deployment detail, not an ongoing number.
// methodology Single-request, warm, tokens-per-second on production-style coder prompts. It measures per-request latency, not concurrent throughput, and uses a word-count proxy for tokens (a slight undercount, but consistent across models). It's honest about its limits — the full write-up and raw runs are in the benchmarks repo so you can check my work.
the part that actually matters: local-first
Speed is the headline; the real reason I run this is everything the tables don't show. Nothing leaves the box. Prompts, context, whatever a security-sensitive workflow feeds the model — it stays on hardware I own, behind my own network. There's no third-party retention policy to read, no token meter running, no rate limit that changes on someone else's roadmap.
And the marginal cost of a run is $0. Once the card is paid for, an agent can loop all night, re-run a failed job, or chew through a long document without a bill materializing. That single fact reorganizes how you build — you stop rationing calls and start treating inference as free compute you already own, which is exactly how the rest of my Dirty South Alpha fleet is designed to work: a memory brain and autonomous agents that lean on a local MoE because leaning on it costs nothing.
Twenty years in security taught me to be suspicious of "just send it to our API." A $500 card that keeps the data home and the meter at zero — running a 30B at 40 tok/s all day — isn't a compromise. For a lot of real work, it's the better architecture. You just have to know that on Arc, the winning 30B is a MoE.
run it yourself
The server, the proven launch configs, and the full benchmark methodology are open — fork them, run your own models, tell me where I'm wrong: