vLLM vs Ollama: run your own office LLM server
vLLM vs Ollama, set up for real: one GPU server in the office or in a data centre in your country, an OpenAI-compatible endpoint, and Locul on top. Real commands, a sizing table by team size, and how to pilot it on rented hardware first.
Quick answer
Use Ollama for a pilot or a small team: install it with curl -fsSL https://ollama.com/install.sh | sh, set OLLAMA_HOST=0.0.0.0:11434 in a systemd override and pull a model. Use vLLM when many people work at once: vllm serve openai/gpt-oss-120b --api-key ... on one 80 GB GPU. Both speak the OpenAI API. Point Locul's Ollama address at the server, and keep port 11434 inside your network.
vLLM vs Ollama: which should run on the office server?
Pick Ollama for a pilot or a team of a few people, and vLLM once many people send prompts at the same time.
Both expose an OpenAI-compatible API, so the apps on top do not care which one you chose. The difference is how they handle a queue of requests and how much setup they ask of you.
| Ollama | vLLM | |
|---|---|---|
| Install | One script, runs as a systemd service | Python package in a virtual environment |
| Model files | GGUF, pulled by name from ollama.com or Hugging Face | Hugging Face weights (safetensors), full or quantised |
| Hardware | NVIDIA, AMD, Apple Silicon, or CPU only | Built for data centre GPUs, NVIDIA first |
| Parallel requests | Default 1 per model, raise with OLLAMA_NUM_PARALLEL | Continuous batching, made for many users at once |
| Sign-in | None built in, keep it behind your firewall | --api-key or VLLM_API_KEY |
| Default address | 127.0.0.1:11434, API under /v1 | localhost:8000, API under /v1 |
| Best for | Pilots, 1 to 10 people, mixed hardware | Busy teams, one big model, steady load |
"vLLM can be deployed as a server that implements the OpenAI API protocol."
vLLM docs, Quickstart
Still weighing whether to run your own model at all? Our self-hosted LLM guide covers build versus buy, and the on-premise LLM guide covers hardware, cost and risk. This page is the hands-on part.
How do I set up an Ollama server for the whole office?
Install Ollama on a Linux box with a GPU, tell it to listen on the network, pull a model and test it with one curl command.
Install
bash$ curl -fsSL https://ollama.com/install.sh | shThe script installs Ollama and registers it as a systemd service, so it starts again after a reboot.
Listen on the office network
By default Ollama only answers on the machine itself. Add an override so other machines can reach it:
bash$ sudo systemctl edit ollama.serviceAdd these lines in the editor that opens, save, then reload:
ini[Service] Environment="OLLAMA_HOST=0.0.0.0:11434" Environment="OLLAMA_NUM_PARALLEL=2"bash$ sudo systemctl daemon-reload $ sudo systemctl restart ollamaPull a model
Pick one from the sizing table. Models on Hugging Face pull straight by name:
bash$ ollama pull hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_MTest it from another machine
bash$ curl http://192.168.1.40:11434/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_M","messages":[{"role":"user","content":"Say hello to the office"}]}'Any JSON reply with a
choiceslist means the server works. A timeout usually means the firewall, not Ollama.Lock it down
Ollama has no sign-in of its own. Allow port 11434 only from your office subnet or VPN, never from the internet:
bash# allow the office subnet only $ sudo ufw allow from 192.168.1.0/24 to any port 11434 proto tcp
"Ollama binds 127.0.0.1 port 11434 by default."
Ollama docs, FAQ
How do I serve a model with vLLM?
Install vLLM into a Python virtual environment, then run vllm serve with the model name and an API key.
$ uv venv --python 3.12 --seed
$ source .venv/bin/activate
$ uv pip install vllm --torch-backend=auto$ vllm serve openai/gpt-oss-120b --api-key "$VLLM_API_KEY"The first start downloads the weights, which takes a while for big models. After that, the OpenAI-compatible API answers on port 8000. Every request must carry the key as Authorization: Bearer.
One model, one GPU. OpenAI says gpt-oss-120b, its open-weight model, fits on a single 80 GB card, because its mixture-of-experts weights ship in MXFP4. That makes it a natural first vLLM model for a team.
"Fit into a single 80GB GPU (like NVIDIA H100 or AMD MI300X)."
OpenAI, gpt-oss-120b model card
How do I point Locul at my own server?
For an Ollama server, type its address into the Locul desktop app. For vLLM, your Locul partner or our team wires it in during the on-premise setup.
Open the local models card
In Locul, open Settings, find the local models card and choose Advanced.
Set the Ollama address
In Ollama address, enter the server and port, for example
192.168.1.40:11434. A bare port number means this machine.Pick the chat model
Under Chat model, choose the model you pulled on the server. Leave the embedding model on automatic unless you have a reason to change it.
Give each bot its model
Bots that handle client files use the office server. A coding bot can still use Claude, if your policy allows it.
vLLM and bigger setups. For a vLLM server, a Team Brain on your own hardware, or several offices, a Locul partner or our team does the wiring during an on-premise deployment. Email us to plan one.
What GPU do I need for my team size?
A 24 GB GPU covers a small office on a 9 to 12 billion parameter model, and a single 80 to 96 GB card runs a 120B-class model for a bigger team.
| Team size | Model class | GPU and VRAM | Server | People at once |
|---|---|---|---|---|
| 1 to 5 | 9 to 12B, Q4 or Q8 (Qwen3.5 9B, Gemma 4 12B) | One 24 GB card | Ollama | One or two; raise OLLAMA_NUM_PARALLEL if VRAM allows |
| 5 to 25 | 27 to 35B, Q4 (Qwen3.8 27B, Qwen3.6 35B-A3B) | 48 GB (L40S) or 96 GB (RTX PRO 6000) | Ollama or vLLM | A handful; queues form at busy moments |
| 25 to 100 | gpt-oss-120b | One 80 GB H100 or a 96 GB card | vLLM | Many; batching keeps the queue moving |
| 100 or more | 120B class, several copies, or larger models | Several GPUs (4x RTX PRO 6000, H100 nodes) | vLLM, tensor parallel | Plan with a partner |
Model sizes on disk come from the files on Hugging Face: a Q4_K_M build of Qwen3.5 9B is 5.68 GB, Gemma 4 12B is 7.12 GB and Qwen3.6 35B-A3B is 22.13 GB. Leave room on the card for the context of every request running at once.
"Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH."
Ollama docs, FAQ
We do not quote tokens per second here. Speed depends on the model, the quantisation, the prompt length and how many people press Enter at once. Our own measurements will go in the benchmark section when they are final.
Can I rent an EU GPU for a pilot first?
Yes. Rent a GPU server from an EU provider for a few weeks, run the same setup as above and decide on hardware afterwards.
| Provider | GPU | VRAM | Where | Good for |
|---|---|---|---|---|
| Scaleway | L40S | 48 GB | Paris, France | 27 to 35B models for a mid-sized team |
| Scaleway | H100 | 80 GB | Paris, France | gpt-oss-120b on vLLM |
| Hetzner GEX45 | RTX PRO 4000 Blackwell | 24 GB | Helsinki, Finland | A small office pilot on Ollama |
| Hetzner GEX131 | RTX PRO 6000 Blackwell Max-Q | 96 GB | Falkenstein, Germany, or Helsinki | 120B-class models with room for context |
Check current rates on Scaleway's GPU pricing page and Hetzner's GPU server page. Both are EU companies with EU data centres. Scaleway GPU Instances are cloud servers you can start and stop, while Hetzner GEX machines are dedicated servers rented by the month.
What does an on-premise model handle well?
The everyday, slightly messy jobs that involve client names and numbers you would rather not send abroad.
Here is a request a partner at a small accounting firm might send to a bot running on the office server:
Everything in that message, the client name, the amounts and the colleague who made the mistake, stays on hardware you control. The bot reads the sheet and the credit note from Locul's memory and answers on the same server.
How fast is an office server in practice?
We are timing real office tasks on the GPUs in the tables above, and the numbers will be published here.
Benchmark in progress
Response times and answer quality per GPU, with one, five and twenty people at once, on Ollama and on vLLM. Coming soon.
Frequently asked questions
Is vLLM faster than Ollama?
With many people sending prompts at once, vLLM usually keeps up better because it batches requests continuously. For one or two people on a single GPU the difference is small, and Ollama is far quicker to set up.
Can Ollama serve several users at once?
Yes. Set OLLAMA_NUM_PARALLEL above its default of 1. Each extra parallel slot needs memory for its own context, so watch VRAM as you raise it.
Does Ollama have authentication?
No. Ollama has no built-in sign-in, so keep it on your office network or VPN and block port 11434 from the internet. vLLM can require an API key with --api-key.
What is the cheapest way to test an on-premise LLM?
Rent a GPU server from an EU provider such as Scaleway or Hetzner for a short pilot, install Ollama, and point Locul at it. Buy hardware only once the pilot shows which model size your team needs.
Do I need a GPU at all?
For a team, yes. Ollama runs on CPU only, but replies become slow enough that people stop using it. A single 24 GB GPU is a sensible floor for a small office.
Sources
Every command and quote on this page comes from these primary sources, checked on 24 October 2026.
Junaid builds Locul and sets up private AI for small regulated firms. He tests every command on this page on his own machines before it ships.