Ollama downloads quantized GGUF models from a registry and serves them over a local HTTP API, handling GPU detection, layer offload and model lifecycle for you. It replaces a paid OpenAI or Anthropic API key for workloads you are willing to run on your own silicon.
Our verdict on Ollama#
Get the memory model right and everything else follows. Ollama loads as many model layers into VRAM as fit and runs the rest on the CPU, so performance is a cliff and not a slope: a model that fits entirely in VRAM is fast, and one that misses by two layers is several times slower. Without a GPU it falls back to a pure CPU path and takes the model out of system RAM. Calibrate on rough numbers: an 8B model at Q4_K_M is about 5 GB and generates on the order of 5 to 15 tokens per second on a modern desktop CPU, which is slower than you read but usable for batch work; a 70B at Q4 is around 40 GB and drops to roughly 1 to 3 tokens per second, which is not usable interactively. Those are approximations that scale with memory bandwidth. The default context is 4096 tokens, which surprises people whose long prompts get silently truncated, and models unload after 5 minutes idle.
Who Ollama is for#
Choose it if
- Anyone with a 12 GB or larger GPU who wants a coding or chat model that never sends a token off the machine.
- Developers wanting a drop-in local endpoint at
localhost:11434for scripts and agents during development. - Apple Silicon owners, where unified memory means a 32 GB Mac can hold models a 12 GB discrete GPU cannot.
- Batch workloads such as summarizing a document pile overnight, where 8 tokens per second is fine.
Look elsewhere if
- You have under 8 GB of RAM and no GPU; you will be running 1 to 3B models that are not good enough for real work.
- You need frontier-model quality, because a 70B quantized local model is not equivalent to a current hosted flagship.
- You want multi-tenant serving with batching and high throughput, which is what vLLM or SGLang exist for.
What running it actually looks like#
One binary or one container. It binds 127.0.0.1:11434 by default and you change that with OLLAMA_HOST, which is the moment you also need to put something in front of it, because there is no auth. On Linux the installer creates an ollama system user and stores models in /usr/share/ollama/.ollama/models; on macOS and Windows they land under the user profile. OLLAMA_MODELS moves that directory, which is what you want when the models belong on a separate disk. GPU support needs the driver on the host and the container toolkit if you are containerized: NVIDIA compute capability 5.0 or higher with a 550-series or newer driver, ROCm v7 for AMD on Linux, Metal on Apple, and Vulkan as a fallback for some Intel and AMD parts. First run pulls several gigabytes before it answers anything.
Resource profile#
- Memory
- 150 MB idle, around 6000 MB in ordinary use. The server itself is a small Go process. Typical here is one 7 to 8 billion parameter model at Q4_K_M resident, which is roughly 5 to 6 GB; without a GPU that comes out of system RAM, with a GPU it comes out of VRAM.
- CPU and acceleration
- CPU-only inference is memory-bandwidth-bound, not core-bound, so a 16-core server with dual-channel DDR4 is barely faster than an 8-core one. A GPU with enough VRAM to hold the whole model is the only change that matters.
- Storage growth
- Models are the entire story. A 7 to 8B model at Q4 is 4 to 5 GB, a 70B at Q4 is around 40 GB, and a casual `ollama pull` habit will fill a 256 GB disk within a month.
- Operational load
- 2 of 5, Light. One process, one directory of blobs, no database, and upgrades are a binary swap. The recurring attention is disk pressure from accumulated models and GPU driver compatibility after a kernel update.
Figures describe a small single-household install and are the working assumptions behind our stack planner. Your numbers will differ with library size, user count and hardware. See methodology.
Data and backup surface#
One directory holds everything: models/blobs for the content-addressed weight files and models/manifests for the tags that point at them. Nothing is a live database, so a file copy is safe even while the server runs. The real question is whether you should back it up at all. Every published model is re-pullable from the registry, so for most people the correct backup is a text file listing ollama list output, and the blobs are excluded. What is not regenerable is anything local: custom Modelfile definitions, models you built with ollama create, and any weights you imported from a GGUF file you no longer have. Keep those, and keep the ~/.ollama/id_ed25519 key if you push to a registry.
Derived backup shape
File copy. Copy the config and data directories. Safe while running for anything that writes atomically.
Traps and surprises#
The API has no authentication
Setting OLLAMA_HOST=0.0.0.0 exposes an unauthenticated API that will pull models, run inference and delete models for anyone who can reach the port. Internet-wide scans for open 11434 endpoints have been running for a while. Keep it on localhost and reach it through a reverse proxy that requires a token, or bind it to a WireGuard interface only.
Default context is 4096 tokens
OLLAMA_CONTEXT_LENGTH defaults to 4096 regardless of what the model supports. Feed a 20,000 token document into a model advertised as 128k and the front of it is silently dropped, producing answers that confidently ignore half the input. Raise it per request with num_ctx or globally with the environment variable, and remember that a large context allocates a proportionally large KV cache in VRAM.
Partial GPU offload is a performance cliff
Ollama offloads as many layers as VRAM allows and runs the remainder on the CPU. Missing the fit by a small margin does not cost a small amount of speed; throughput typically drops several times over because every token now crosses the PCIe bus. Watch the layer count in the server log, and drop to a smaller quantization or a shorter context so the whole model fits rather than accepting a partial offload.
Models unload after five minutes
The default keep-alive is 5 minutes, after which the model is evicted and the next request pays the full load time again, which is tens of seconds for a large model off a spinning disk. For an always-on assistant set OLLAMA_KEEP_ALIVE to a long duration or a negative value to pin it, and accept that the VRAM stays occupied. Conversely, OLLAMA_MAX_LOADED_MODELS defaults to 3, so three different models can quietly fill your GPU.
Quantization level is not a detail
The default tags in the library are typically Q4_K_M, a four-bit quantization that trades measurable quality for roughly half the memory of Q8. Pulling :latest therefore gets you the compromise, not the model as published. If you have the VRAM, Q6 or Q8 tags are noticeably better on reasoning and code; if you are memory-starved, Q3 variants exist and degrade badly. Check the tag list rather than assuming.
Specifications#
- Category
- Local AI and private search
- Licence
- MIT (Permissive)
- Written in
- Go / C++
- First release
- 2023
- Maturity
- Stable
- Datastore
- None; a content-addressed blob store on disk
- Default ports
11434- Ships as
- docker, native binary, macOS app, Windows installer
- arm64 builds
- Yes
- Identity
- Not applicable. The API has no authentication of any kind; it is designed to listen on localhost and anything beyond that is your reverse proxy's problem.
- Replaces
- OpenAI API, Anthropic API, GitHub Copilot, ChatGPT Plus
- Project site
- ollama.com
- Source
- github.com/ollama/ollama
- Documentation
- docs.ollama.com
Alternatives to Ollama#
Everything else in local ai and private search, closest in operational weight first.
| Tool | Ops load | Idle RAM | Licence |
|---|---|---|---|
| Open WebUI | 2, Light | 600 MB | Open WebUI License (BSD-3-Clause plus a branding clause) |
| SearXNG | 2, Light | 150 MB | AGPL-3.0-or-later |
| LocalAI | 3, Moderate | 300 MB | MIT |
Where Ollama comes up elsewhere#
Head to head
Sources#
Facts on this page were checked against the project's own material. Where a number is our own estimate rather than a published figure, the text says so.
Published . Last reviewed . Found something out of date? Tell us and we will fix it and log the change.