Ollama vs llama.cpp vs vLLM for Local AI Development

Local model serving gets messy fast. A setup that feels great on a laptop can become the wrong choice the moment you need structured outputs, higher concurrency, or a clean OpenAI-compatible endpoint for tools.

The mistake I see most often is treating Ollama, llama.cpp, and vLLM as interchangeable. They are not. All three can run a model locally, but they optimize for different things: convenience, low-level control, and throughput.

If I were setting up a practical local AI stack for real developer work, I would choose based on deployment shape first, not benchmark screenshots.

Why this matters

If your local AI runtime sits behind coding tools, eval jobs, or internal automation, the serving layer becomes part of the product. Token speed matters, but so do API compatibility, memory behavior, quantization support, batching, startup time, and debuggability.

• the model fits, but latency is painful
• the model is fast, but the API shape is awkward for tools
• concurrency looks fine in a demo, then collapses under parallel requests
• the system works until you switch hardware or model format

Architecture and workflow overview

flowchart TD
    Need local model serving --> Hardware{What hardware do you have?}
    Hardware -->|CPU or Apple Silicon| Simplicity{Need simple setup?}
    Simplicity -->|Yes| Ollama[Choose Ollama]
    Simplicity -->|No| Llama[Choose llama.cpp]
    Hardware -->|NVIDIA GPU server| Scope{Single-user or shared service?}
    Scope -->|Single-user| O2[Ollama can be enough]
    Scope -->|Shared service| V[vLLM]

Implementation details

Ollama, the easiest way to get to working

Ollama is what I reach for when I want a usable endpoint in minutes. It hides a lot of model and runtime complexity behind a predictable CLI and API.

curl -fsSL https://ollama.com/install.sh | sh
ollama serve
ollama pull qwen2.5-coder:14b
ollama run qwen2.5-coder:14b

curl http://localhost:11434/api/generate \
  -d '{
    "model": "qwen2.5-coder:14b",
    "prompt": "Write a Python function that validates a webhook signature.",
    "stream": false
  }'

llama.cpp, when hardware reality matters more than polish

llama.cpp is the runtime I trust when I care about exact control. It is especially useful when I need GGUF models, aggressive quantization, or a setup that runs acceptably on weaker hardware.

./llama-server \
  -m ./models/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf \
  -c 8192 \
  -ngl 999 \
  --host 0.0.0.0 \
  --port 8080

curl http://localhost:8080/completion \
  -H 'Content-Type: application/json' \
  -d '{
    "prompt": "Summarize the security risks of exposing a local model server to a LAN.",
    "n_predict": 220,
    "temperature": 0.2
  }'

vLLM, when local becomes a real service

vLLM is the one I pick when the workload starts looking like infrastructure instead of a personal toy.

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-Coder-14B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --max-model-len 16384

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local-dev")
resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-Coder-14B-Instruct",
    messages=[
        {"role": "system", "content": "You are a careful coding assistant."},
        {"role": "user", "content": "Generate a bash script that rotates nginx logs safely."}
    ],
    temperature=0.1,
)
print(resp.choices[0].message.content)

What went wrong and the tradeoffs that actually matter

Quantization is not free, API compatibility becomes a force multiplier, oversized context defaults waste memory fast, and local model servers stop being local the second you bind them to a wider network.

Terminal-output style reality check

$ ollama ps
NAME                    ID              SIZE      PROCESSOR    UNTIL
qwen2.5-coder:14b       8f3c1f2f3d44    9.0 GB    100% GPU     4 minutes from now

$ curl -s http://localhost:8000/v1/models | jq '.data[0].id'
"Qwen/Qwen2.5-Coder-14B-Instruct"

$ ./llama-bench -m ./models/qwen-coder-q4.gguf -ngl 999
prompt eval time =  412.14 ms / 128 tokens
generation time  = 6150.52 ms / 256 runs   (24.00 tok/s)

Best-practice checklist

Conclusion

Ollama, llama.cpp, and vLLM all belong in a serious local AI toolkit. For most solo developer laptops, I would start with Ollama. For constrained hardware or GGUF-heavy experimentation, I would reach for llama.cpp. For shared GPU-backed services, I would move to vLLM quickly.