Run vLLM on the DGX Spark

You want batched, paged-attention serving with better concurrency than Ollama gives you. vLLM runs an OpenAI-compatible server (default port 8000) and is the right tool once you are past “one chat at a time” and into several callers, benchmarks, or production-shaped load.

The catch on this box is the silicon, not vLLM itself.

Why a plain pip install vllm fails

The GB10 GPU is sm_121a, which only CUDA 13 supports. Almost every published Python wheel (vLLM, flash-attn, and friends) is built against CUDA 12.x, so importing one on the Spark dies with:

ImportError: libcudart.so.12: cannot open shared object file

There is no CUDA 12 on the box to satisfy that. So the reliable paths are a container that already targets CUDA 13, or a cu130 build. Reach for a container first.

Caution

Memory here is unified, so a GPU out-of-memory is not a contained crash. It is system-wide memory exhaustion that can freeze the whole machine, not just kill the server. Treat --gpu-memory-utilization as a safety belt: start around 0.85 and bound the workload with --max-model-len and --max-num-seqs before you push it higher.

Serve with a container

1. Pick an image. Both expose vLLM’s OpenAI server on :8000.

   NGC (supported baseline)

   NVIDIA publishes a vLLM image on NGC built for the Spark. It is the supported path and tends to lag upstream vLLM by a release or two.

   docker run --gpus all --rm -p 8000:8000 --ipc=host \
     nvcr.io/nvidia/vllm:<tag> \
     --model Qwen/Qwen3-32B \
     --gpu-memory-utilization 0.85

   Community (latest vLLM)

   If you need a current vLLM release, timothystewart6/vllm-gb10 tracks upstream and is built on a Spark for sm_121a.

   docker run --gpus all --rm -p 8000:8000 --ipc=host \
     timothystewart6/vllm-gb10:latest \
     --model Qwen/Qwen3-32B \
     --gpu-memory-utilization 0.85

   The vllm/vllm-openai family uses vllm serve as its entrypoint, so in a docker-compose.yml the command: passes only the model and flags, never vllm serve again.

2. Confirm it is serving. The API is OpenAI-shaped, same as the Ollama recipe:

   curl http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen/Qwen3-32B","messages":[{"role":"user","content":"Hello"}]}'

Good for: several concurrent callers, benchmarking, throughput-shaped serving, and feeding a gateway that sits in front of multiple apps.

Other install paths

Containers are the low-friction route. The alternatives trade convenience for control:

• Pinned cu130 nightly wheels. The fastest no-compile native install: a uv venv, PyTorch from the cu130 index, then uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly/cu130. Honest caveat: there is no stable cu130 aarch64 vLLM yet, so you are pinning a nightly that can stop being hosted.
• From source. A 20 to 30 minute compile targeting sm_121a. Heaviest, but you know exactly what went into the build and can pin it.
• Nix devshell. nix develop github:graham33/nixos-dgx-spark#vllm-nix gives a reproducible environment without committing the whole box to NixOS.

Don’t fight the kernels

Two things that look like problems but are not:

• Skip flash-attn. vLLM bundles its own FlashInfer kernels, and the upstream flash-attn pip package fails to load here anyway. PyTorch’s native SDPA with cuDNN is actually faster on Blackwell.
• Ignore the capability warning. You will see Found GPU0 NVIDIA GB10 which is of cuda capability 12.1. Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0). It is harmless: sm_120 and sm_121 are binary-compatible, so sm_120 kernels run fine.

Sizing and going multi-node

The weights plus KV cache still have to fit the 128 GB pool. The model sizing table applies here too, and unified memory explains why the whole pool is genuinely usable. For more than one box, distributed vLLM (with Ray) runs tensor-parallel across two Sparks over the fast fabric. See connect two Sparks and the NCCL build it depends on.

Serve a local LLM The other serving options (Ollama, llama.cpp) and the model sizing table.

Specifications Memory, compute, and model-capacity facts for sizing a deployment.

Connect two Sparks Distributed, tensor-parallel vLLM across two units over 200GbE.

Playbooks NVIDIA's first-party vLLM, TensorRT-LLM, and SGLang playbooks.