Model throughput, measured

The Spark’s memory bandwidth is roughly 273 GB/s, which is modest. That number gets quoted as “the Spark is slow”, and for dense models it is true. It is also the wrong lesson to draw, because it does not generalise to the architecture that actually suits this machine.

Decode speed is bandwidth-bound: generating each token requires reading the weights that participate in that token. For a dense model that is every weight. For a mixture-of-experts model it is only the active subset. Speed tracks active parameters, not total parameters, and total parameters are what memory capacity constrains.

A box with lots of memory and modest bandwidth is therefore close to the ideal MoE host, and close to the worst dense host.

Measured, same prompt, warm

Model	Architecture	Total	Active	Resident	tok/s
qwen3:32b	dense	32.8B	32.8B	64 GB @ 40k	10.0
qwen3.5:122b	MoE	125B	not published	81 GB @ 32k	29.0
qwen3-coder:30b	MoE	30.5B	~3.3B	45 GB @ 64k	69.3

The ordering is the point. The largest model is three times faster than the smallest one, and the fastest model is the one with the fewest active parameters, not the fewest total parameters.

Two consequences worth internalising:

The default chat model is the worst architectural fit on the box. qwen3:32b is dense, so it pays full bandwidth cost for every token and lands at 10 tok/s. Nothing is misconfigured. It is simply the wrong shape of model for this hardware.

The coder model was already seven times faster than the chat model and nobody had measured it. qwen3-coder:30b is a 30B MoE with roughly 3.3B active, which is why it feels responsive in a way the chat tier does not.

Capacity is the real constraint, not speed

qwen3.5:122b holds 81 GB of weights. Dropping its context from 256k to 32k only recovered about 7 GB, because weights dominate and the KV cache does not. That sets a hard floor: at 81 GB it leaves roughly 40 GB for everything else, which is enough for the embedding service but not for a second large model.

So the choice is not simply “pick the fastest”. It is a packing problem. A 50 GB MoE that leaves room for a companion model may serve the fleet better than an 81 GB one that does not, even if their token rates are similar.

Reasoning models and the think:false trap

Disabling thinking can silently produce wrong answers

qwen3.5:122b is a reasoning model. Our aliases historically pass think:false so that plain text lands in content for callers that do not understand reasoning_content. On a reasoning model that flag is not a formatting preference, it removes the mechanism the model uses to be correct.

Same arithmetic word problem, same model, one flag apart:

think:false → "15. ...doubled that amount to get 18, and then subtracted
               the 6 sold to leave him with 15."      ← 18 − 6 ≠ 15
thinking on → "21. After keeping 9 sheep, he buys 18 more to reach 27,
               then subtracts the 6 sold for a final total of 21."

The think:false answer is not merely wrong, it is internally inconsistent: the stated arithmetic does not produce the stated result. Nothing in the response signals degraded output. A dense non-reasoning model of half the size answered this correctly.

The cost of thinking is real but modest here, 24.9 tok/s against 29.0, plus the reasoning tokens themselves. Pay it. If a caller genuinely cannot handle reasoning output, give it a non-reasoning model rather than a lobotomised reasoning one.

How to measure

curl -s http://localhost:11434/api/chat -d '{
  "model": "MODEL", "stream": false, "keep_alive": "3m",
  "options": {"num_ctx": 32768},
  "messages": [{"role":"user","content":"Write a haiku about each of the four seasons. Then count from 1 to 40."}]
}' | jq '{tok_per_s: (.eval_count/(.eval_duration/1e9)), eval_tok: .eval_count}'

Always measure warm. A cold run folds a 30-second-plus weight load into total_duration and tells you about disk, not inference. load_duration near zero confirms the model was already resident. Use a prompt long enough to generate a couple hundred tokens; a two-token reply divides by noise.