SigLIP 2 cross-modal embeddings

SigLIP 2 puts images and text in the same 1152-dimensional space, which is the whole reason it earns a slot here. A text-only embedder can tell you which listing descriptions resemble each other. SigLIP lets you embed the phrase “granite countertops” and rank photographs against it, with no captioning step in between.

On this box it runs as a standalone FastAPI service on port 8002, entirely separate from Ollama. It is reached through the the inference gateway gateway as the model names siglip2, siglip, and google/siglip2-so400m-patch16-naflex.

curl -X POST https://my-app.the inference gateway/v1/embeddings \
  -H "Authorization: Bearer $GPU_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"siglip2","input":"granite countertops"}'

Swap the input for a data:image/... URL, or a {"image_url": {...}} object, and you get a vector in that same space. Compare with a dot product; the service L2-normalises its output, so dot product is cosine similarity.

What it costs the box

2.4 GB of GPU memory. That is the number that decided the move. The Spark’s 128 GB is mostly spoken for by whichever large model Ollama has resident (qwen3:32b sits around 60 GB with four parallel slots), so anything sharing the GPU has to justify its footprint. SigLIP asks for roughly two percent of the machine and then holds steady, because the model is loaded once at startup and never swapped.

It is worth being precise about why this coexists peacefully rather than just noting that it does. SigLIP is not an Ollama model. It never enters Ollama’s scheduler, so it cannot be evicted by a large model loading, and it cannot trigger an eviction itself. The two systems share physical memory and nothing else. That independence is the reason a service this small is safe to colocate with a tier that has a history of memory-pressure wedges.

The migration trap: silently invalidated vectors

The service previously ran on an Apple Silicon laptop, using PyTorch’s MPS backend. Moving it here means the same model weights now execute through CUDA on Blackwell instead. Different silicon, different kernels, different floating-point accumulation order.

This matters more than it first appears. Consumers of this service store their embeddings. A production image search had thousands of vectors already computed on the MPS backend and sitting in a database. If the CUDA backend produced meaningfully different vectors for the same input, every stored vector would silently become incomparable with every new query. Nothing raises an error in that scenario. Search quality just quietly degrades, and the cause is invisible at the API layer, because both backends happily return a well-formed 1152-dim float array.

So the cutover was gated on a parity check rather than a health check. Identical inputs went to both backends while both were still running, and the outputs were compared element by element:

Input	cosine(MPS, CUDA)	max abs delta
image: red circle	0.999765	0.001022
image: blue square	1.000050	0.000366
text: “granite countertops”	1.000162	0.000244
text: “a red circle”	1.000246	0.000244

Agreement to about one part in a thousand, which is fp16 rounding noise and nothing more. The cosine figures slightly above 1.0 are the same rounding showing up in vectors that were normalised before quantisation, not an error.

The number that actually answers the operational question is this one: an image vector embedded on the old backend, scored against a text query embedded on the new one, still ranks correctly.

MPS-image(red circle) vs CUDA-text("a red circle")        0.1794
MPS-image(red circle) vs CUDA-text("granite countertops") -0.0347

Stored vectors remained valid. No re-embedding run was needed.

Rank, don’t threshold

Those similarity values look low if you are used to cosine scores from a text embedder. SigLIP is trained with a sigmoid loss, and good matches routinely land in the 0.10 to 0.20 range. Use the scores to order candidates. An absolute cutoff tuned against some other model’s intuition will throw away correct results.

Device selection

The service picks its accelerator at import time, preferring CUDA, falling back to Apple Silicon MPS, then CPU. SIGLIP_DEVICE overrides it, which is mainly useful for forcing CPU to reproduce a numerical question.

def _pick_device() -> str:
    if forced := os.getenv("SIGLIP_DEVICE"):
        return forced
    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"

/health reports which one it actually got, and checking that field is the fastest way to catch the failure mode where a torch upgrade quietly drops Blackwell support and the service falls back to CPU. It would still answer requests. It would just be slow, and nothing else would tell you why.

Installing torch on Blackwell

This is the one genuinely fiddly part of the deployment. The GB10 reports compute capability 12.1, and the default PyPI torch wheel ships no kernels for it. Installing torch the ordinary way produces a package that imports cleanly, reports torch.cuda.is_available() == True, and then fails when it tries to run an actual kernel.

The fix is to pull torch from NVIDIA’s CUDA 13 index, gated by platform marker so the Macs keep getting the MPS wheel from PyPI:

[[tool.uv.index]]
name = "pytorch-cu130"
url = "https://download.pytorch.org/whl/cu130"
explicit = true

[tool.uv.sources]
torch = [{ index = "pytorch-cu130", marker = "sys_platform == 'linux'" }]

Verify with an actual matrix multiply, not just an availability check:

.venv/bin/python -c "
import torch
print(torch.__version__, torch.cuda.get_device_capability(0))
x = torch.randn(1000, 1000, device='cuda', dtype=torch.float16)
print('kernel ok:', (x @ x).sum().item() is not None)"
# 2.13.0+cu130 (12, 1)
# kernel ok: True

Running it

The service runs under systemd as siglip, enabled at boot, restarting on failure. It runs as a normal user account so it shares that user’s HuggingFace cache rather than re-downloading three gigabytes of weights under a system account.

systemctl status siglip
journalctl -u siglip -f
curl -s http://localhost:8002/health

TimeoutStartSec is set generously because a cold start has to load weights before it binds the port. A short timeout would have systemd declaring failure on a service that was merely still starting.

Check which process owns the port

systemctl is-active returning active does not prove the systemd unit is the process serving traffic. The unit type here reports success once the process execs, not once it binds. During this deployment a manually started copy kept hold of port 8002 after its parent was killed, so the systemd unit crash-looped 38 times against “address already in use” while health checks kept passing, answered by the orphan. Each failed attempt loaded 2.4 GB onto the GPU before dying.

Verify ownership explicitly:

systemctl show siglip -p MainPID -p NRestarts
sudo ss -tlnp | grep 8002    # the PID here must match MainPID

A non-zero, climbing NRestarts on a service that appears healthy is the tell.

Related

• Kokoro text to speech — the other standalone service that moved here, and the easier case: pure CPU, no GPU contention.
• Ollama and model residency — why the large models need watching and this one does not.