Ollama vs LocalAI

The comparison people expect is "which one is faster". It is the wrong axis, because underneath both of these sits llama.cpp doing the same arithmetic on the same weights. On identical hardware with an identical quantization you will not measure a difference worth choosing on.

The real axis is scope. Ollama is a curated single-purpose model runner: pull a model by name, get an HTTP API, done. LocalAI is a gateway that presents the OpenAI API surface and dispatches each request to whichever of dozens of backends serves that model, covering chat, embeddings, audio transcription, speech synthesis, image generation and reranking.

If you just want a local chat model, install Ollama. That is the recommendation, and the rest of this page is about when it is wrong.

What the spec table is actually deciding#

Idle RAM (150 MB against 300 MB) is noise, because both numbers are dwarfed by model weights the moment you load anything: budget 6 GB or more of typical usage for either with a single 8B model resident. The rows that matter are ops load, 2 against 3, and default port, 11434 against 8080, which collides with rather a lot of other software. Everything else that distinguishes them is not a specification, it is a shape.

The complexity difference, quantified#

Ollama's setup, end to end:

docker run -d -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama pull llama3.1:8b
curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"hello"}'

One image tag for CPU and NVIDIA, plus a ROCm variant for AMD. Models land in a content-addressed blob store under /usr/share/ollama/.ollama/models on Linux, and OLLAMA_MODELS moves that directory when the weights belong on a separate disk.

LocalAI's setup involves three decisions Ollama does not ask you to make:

1. Which image variant. Plain CPU, CUDA 12, CUDA 13, ROCm, Intel oneAPI, Vulkan or Jetson L4T. Compressed sizes span an order of magnitude: about 280 MB for CPU, roughly 260 MB on arm64, about 3.5 GB for CUDA 12, over 4 GB for the Intel GPU build.
2. Which backends to install. Recent LocalAI moved backends out of the image into a runtime gallery, which is why the base image is small. Mount /backends as a volume or every container recreation re-downloads them, and on an air-gapped host the model simply fails with a missing-backend error.
3. The per-model YAML. Each model gets a file naming its backend, prompt template, context size and parameters.

You need four volumes rather than one: /models, /backends, /configuration and /data.

The one genuinely irreplaceable thing in a LocalAI install is that YAML. Weights are re-downloadable and backends are re-installable, so backing up /models/*.yaml plus /configuration covers the work you actually did.

VRAM, system RAM and quantization: get this right first#

This is the part that determines whether local inference is pleasant or pointless, and it applies identically to both tools.

Model weights have to live somewhere. With a GPU, they live in VRAM and the GPU does the arithmetic. With no GPU, the same weights come out of system RAM and the CPU does it, roughly an order of magnitude slower. Apple Silicon blurs this: unified memory means a 32 GB Mac holds models a 12 GB discrete card cannot.

Quantization is how you make weights smaller. The default tags in Ollama's library are typically Q4_K_M, a four-bit quantization that uses roughly half the memory of Q8 and gives up measurable quality on reasoning and code. Pulling :latest gets you that compromise, not the model as published. If you have VRAM to spare, the Q6 and Q8 tags are noticeably better; if you are memory-starved, Q3 variants exist and degrade badly.

Rough arithmetic to size a card, about 0.6 GB per billion parameters at Q4, plus KV cache for the context window:

The rule that matters more than any of these numbers: the whole model must fit. Ollama offloads as many layers as VRAM allows and runs the remainder on CPU, and missing the fit by a small margin does not cost a small amount of speed. Throughput typically drops several times over, because every token now crosses the PCIe bus. Watch the layer count in the server log, and drop to a smaller quantization or a shorter context rather than accepting a partial offload. LocalAI has the same failure with a quieter symptom: the wrong image variant against your driver starts fine, loads the model and runs entirely on CPU.

Two Ollama defaults that consume VRAM without telling you: OLLAMA_MAX_LOADED_MODELS is 3, so three different models can quietly fill a card, and models unload after 5 minutes idle, so an always-on assistant needs OLLAMA_KEEP_ALIVE set long or negative to avoid paying tens of seconds of load time on every request.

Honest speed expectations without a GPU#

These are approximations that scale with memory bandwidth, not guarantees, and they are bound by bandwidth rather than core count. A 16-core server on dual-channel DDR4 is barely faster than an 8-core one.

• 8B at Q4_K_M on a modern desktop CPU: roughly 5 to 15 tokens per second. Slower than you read. Fine for batch work: summarizing a folder of documents overnight, tagging a bookmark archive, generating draft text you will edit.
• 70B at Q4 on the same CPU: roughly 1 to 3 tokens per second. Not usable interactively. A paragraph takes minutes.
• Under 8 GB of RAM with no GPU: you are running 1 to 3B models, and they are not good enough for real work. Do not buy hardware on the assumption this will feel like a hosted model.

That last point is the uncomfortable one. A 70B quantized local model is not equivalent to a current hosted flagship, and no amount of runtime choice changes that. Local inference buys you privacy, offline operation and no per-token bill. It does not buy you frontier quality. GPUs, transcoding and local AI covers what hardware actually moves the needle, and What a home server costs to run covers what leaving a GPU resident costs per month.

Neither one is safe on a network#

Ollama has no authentication of any kind. LocalAI has a list of static API keys checked against the Authorization header, with no rotation and no per-key scoping. Both are designed to sit on localhost.

The correct pattern for either: bind to loopback, then front it with a reverse proxy that requires a credential, or expose it only on a WireGuard or Tailscale interface. Reverse proxy and TLS has the Caddy configuration, Remote access without port forwarding covers the VPN route, and A security baseline for a home server is worth a read before you set OLLAMA_HOST=0.0.0.0 on anything with a public IP.

The front end is not part of this decision#

Open WebUI talks to Ollama's native API and to any OpenAI-compatible endpoint, which includes LocalAI. So the chat interface is orthogonal, and you should size it separately: it idles around 600 MB, four times the Ollama server process. On a box that also has to hold model weights, that is a real allocation. Other consumers of these endpoints, such as the LLM tagging in Karakeep, point at whichever URL you give them.

The verdict, by situation#

If you are unsure, the decision is easy: Ollama takes ten minutes and roughly 5 GB of disk to prove out. If you later need transcription or image generation, adding LocalAI beside it is fine, and running both is common. What is not fine is choosing LocalAI first, spending an afternoon on the CUDA variant and the prompt template, and only then discovering that all you wanted was a chat box.

What to do next#

Start by checking what will actually fit: nvidia-smi for VRAM, or free -g if you have no GPU, then pick a model from the table above rather than pulling the biggest thing available. Bind to localhost, put Open WebUI in front, and raise OLLAMA_CONTEXT_LENGTH before you feed it anything long. Then size the whole thing against your hardware in Stack planner, and see Replacing ChatGPT for what a local setup does and does not replace. The rest of the category is in Local AI, including SearXNG if private search is the other half of what you are building.