Ollama and model residency

Ollama runs under systemd here, with an override that shapes how much memory every loaded model consumes:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_MODELS=/usr/share/ollama/.ollama/models"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_FLASH_ATTENTION=1"

OLLAMA_NUM_PARALLEL=4 is the line that matters most, and it is the one easiest to read past. It allows four concurrent requests against a loaded model, which is exactly what you want on a machine with this much memory. What it also does is allocate four independent KV caches, one per slot.

Residency is not file size

A model’s memory footprint is its weights plus its KV cache, and the KV cache scales with context length times parallel slots. The weights are the part you can see on disk. The rest is invisible until the thing is running.

qwen3-coder:30b is an 18 GB file. Its published Modelfile sets a default context of 262144 tokens. Multiply that context against four parallel slots and the arithmetic stops being cute:

$ curl -s http://localhost:11434/api/ps | jq -r '.models[] | "\(.name) \(.size_vram/1e9|round)GB ctx \(.context_length)"'
qwen3-coder:30b 123GB ctx 262144

122 GB resident for an 18 GB model. On a 128 GB box that is not a performance problem, it is a total-loss problem: the first request for this model evicted everything else and left nothing for anything that arrived next.

Pinning context at the gateway

The fix is to pin num_ctx to something a real workload needs, in the gateway’s per-model parameters rather than on the box:

- model_name: qwen3-coder:30b
  litellm_params:
    model: ollama_chat/qwen3-coder:30b
    api_base: http://your-spark:11434
    keep_alive: "5m"
    extra_body:
      options:
        num_ctx: 65536

64k is still a generous coding context, and it lands the model at 45 GB resident. That leaves room for the default chat model to stay loaded alongside it:

qwen3:32b       64GB  ctx 40960
qwen3-coder:30b 45GB  ctx 65536

Two large models, coexisting, on a box where the predecessor could hold one.

Pin it everywhere or nowhere

Sending num_ctx on some requests is worse than never sending it. Ollama reloads the model when the requested context differs from the resident one, so a client that intermittently passes num_ctx triggers a full unload and reload of tens of gigabytes on those requests. We have watched a single misbehaving chat client do exactly this and take a backend down with what looked like random timeouts.

Pinning at the gateway is safe precisely because it is unconditional. Every request carries the same value, so the resident model always matches what is being asked for and no reload is ever triggered. The danger is in the mismatch, not the parameter.

keep_alive strategy

OLLAMA_KEEP_ALIVE=-1 on the box means models stay loaded indefinitely by default. The gateway then overrides this per model, which inverts the decision in a useful way: the box is permissive, and the routing layer decides what deserves to squat in memory.

Hot paths get 30 minutes. Large secondary models get 5, so they release promptly instead of holding 45 GB hostage after a single request. This matters less than it did on the old 48 GB machine, where every large model load was an eviction, but it still matters, because two large models plus a third arriving is still one too many.

Checking what is actually resident

# what is loaded, how big, and at what context
curl -s http://localhost:11434/api/ps | jq '.models[] | {name, size_vram, context_length}'

# per-process GPU memory, which includes non-Ollama services
nvidia-smi --query-compute-apps=pid,used_memory --format=csv

The context_length field in /api/ps is the one to check after any change to a model’s parameters. It reports what the model actually loaded with, which is the ground truth, as distinct from what you believe you configured.