Category
Local AI: VRAM is the constraint, everything else is detail
One number decides whether local models are useful to you or a novelty, and it is not the CPU, the RAM total or the model name. It is how much fits in VRAM.
One number decides this#
Everything else in local AI is detail. The question is how many gigabytes of model fit in memory the accelerator can reach, and the arithmetic is short: parameters times bits, divided by eight, plus a KV cache that grows with your context length.
A 7 to 8 billion parameter model at Q4_K_M is about 5 GB. A 70B at Q4 is about 40 GB. A 12 GB card holds the first with a long context, a 24 GB card reaches the 30B class, and the 70B needs two 24 GB cards or a large unified-memory Mac, which is why Apple Silicon punches above its price here.
Partial offload is a cliff, not a slope. Ollama loads as many layers into VRAM as fit and runs the rest on the CPU, so missing by two layers does not cost you a little speed: every token now crosses the PCIe bus and throughput drops several times over. Watch the layer count in the server log, then drop a quantization level or shorten the context until the whole model fits.
With no GPU at all, you are on the pure CPU path, taking the model out of system RAM, and the limit is memory bandwidth rather than cores. Expect roughly 5 to 15 tokens per second for an 8B at Q4 on a modern desktop CPU: slower than you read, fine for overnight batch work, frustrating for chat. A 70B falls to roughly 1 to 3 tokens per second and is not usable interactively. Under 8 GB of RAM with no GPU you are limited to 1 to 3B models, which are not good enough for real work. That is not a tuning problem and there is no configuration that fixes it.
The default#
Ollama behind Open WebUI. Ollama handles GPU detection, layer offload and model lifecycle, serves an API on 11434, and idles near 150 MB before a model loads. Open WebUI adds accounts, history, group permissions and document retrieval for roughly 600 MB idle, or less on main-slim, which drops the bundled embedding and speech models for about 0.3 GB smaller pull.
Two costs. Ollama's API has no authentication of any kind, and internet-wide scans for open 11434 endpoints have been running for a while. And Open WebUI's licence has not been plain BSD since v0.6.6 in April 2025; the branding clause exempts deployments with 50 or fewer users in a rolling 30-day window, which covers a household and does not cover a 200-person company portal.
The default is wrong in two cases. You need more than chat behind one OpenAI-compatible endpoint, meaning transcription, speech synthesis, image generation or reranking: LocalAI, choosing the image variant that matches your accelerator. You have no GPU and under 8 GB of RAM: skip the model runners entirely and take the other half of this category. SearXNG gives you private metasearch in about 150 MB with no accelerator at all, and it is the more useful of the two on that hardware.
The mistakes#
- Publishing Ollama.
OLLAMA_HOST=0.0.0.0exposes an unauthenticated API that will pull, run and delete models for anyone who reaches the port. Keep it on localhost, or bind it to a WireGuard interface, and put a token-checking proxy in front. - Ignoring the 4096 token default context.
OLLAMA_CONTEXT_LENGTHis 4096 regardless of what the model advertises. Feed in a 20,000 token document and the front of it is silently dropped, producing confident answers that ignore half the input. - Running a CUDA image against the wrong driver. LocalAI starts, loads a model and silently falls back to CPU, so the only symptom is that it is slow. Check the startup log for which backend actually initialized.
- Not persisting LocalAI's
/backendsvolume. Backends are installed at runtime from a gallery, so every container recreation re-downloads them, and on an air-gapped host the model just fails to load. OLLAMA_BASE_URL=http://localhost:11434inside Open WebUI. That is the container's own loopback. The symptom is an empty model list with no error. Usehost.docker.internalor the compose service name.- Editing environment variables after Open WebUI's first boot. Persisted config wins: settings are copied into the database on first launch and read from there. Change it in the admin UI, or set
ENABLE_PERSISTENT_CONFIG=False. - Enabling SearXNG's JSON format on a reachable instance. It is off by default. Turn it on and leave the instance open and you have published an unauthenticated search API that strangers will use through your IP.
Read next#
GPUs, transcoding and local AI has the card-by-card hardware detail and the transcoding overlap. Ollama vs LocalAI settles the runner. Replacing ChatGPT and Replacing Google Search cover what you actually give up. Before you buy anything, price the idle draw in What a home server costs to run.
Every local ai tool we profile#
Sorted by operational load, lightest commitment first. Every row links to a full profile with the resource numbers, the upgrade traps and the restore path.
| Tool | Ops load | Idle RAM | Licence | Identity | Backup shape |
|---|---|---|---|---|---|
| Ollama | 2, Light | 150 MB | MIT | Not applicable | File copy |
| Open WebUI | 2, Light | 600 MB | Open WebUI License (BSD-3-Clause plus a branding clause) | Native OIDC | SQLite backup |
| SearXNG | 2, Light | 150 MB | AGPL-3.0-or-later | Not applicable | File copy |
| LocalAI | 3, Moderate | 300 MB | MIT | Not applicable | File copy |
Profiles#
Ollama
Pull and run quantized language models locally with one command and an HTTP API
ops 2/5 · 600 MBOpen WebUI
Chat front end for local and remote models, no longer under a plain BSD licence
ops 2/5 · 150 MBSearXNG
Metasearch front end that queries other engines for you and stores no profile
ops 3/5 · 300 MBLocalAI
One OpenAI-compatible API in front of many inference backends, not just text
Head to head in this category#
- Ollama vs LocalAIOne command and a chat model, or one API in front of chat, speech, images and embeddings.
Leaving a cloud service in this category#
- Replace ChatGPT: run a chat model on your own hardwareA local model is not a frontier model. Here is the capability gap in numbers, then the case for running one anyway.
- Replace Google Search with your own metasearchSearXNG has no crawler. It asks the same engines you already use, and they can tell it is a server.
Guides for this category#
- GPUs, transcoding and local AI: what hardware you actually needQuick Sync handles your 4K streams for 10 W. A model that fits in VRAM is a different purchase entirely.
Questions#
How much VRAM do I need for local AI?
Work from the weights. A 7 to 8 billion parameter model at Q4_K_M is roughly 5 GB, and the KV cache for a long context sits on top of that. So 8 GB is tight, 12 GB is comfortable for an 8B with real context, and 24 GB gets you into the 30B class at four-bit. A 70B at Q4 is around 40 GB, which means two 24 GB cards or a large unified-memory Mac. Apple Silicon inverts the usual advice: a 32 GB Mac holds models that a 12 GB discrete GPU cannot touch.
Can I run local AI with no GPU at all?
Yes, and you should calibrate your expectations in tokens per second. An 8B model at Q4_K_M generates on the order of 5 to 15 tokens per second on a modern desktop CPU, which is slower than you read but perfectly usable for overnight batch work like summarizing a document pile. A 70B at Q4 drops to roughly 1 to 3 tokens per second, which is not usable interactively. CPU inference is memory-bandwidth-bound, not core-bound, so a 16-core box with dual-channel memory is barely faster than an 8-core one.
What does quantization actually cost me?
The default tags in most libraries are Q4_K_M, a four-bit quantization that roughly halves the memory of Q8 in exchange for measurable quality loss. Pulling :latest therefore gets you the compromise rather than the model as published. If you have the VRAM headroom, Q6 or Q8 tags are noticeably better on reasoning and code. If you are memory-starved, Q3 variants exist and degrade badly. Check the tag list rather than assuming, and remember that raising the context length allocates a proportionally larger KV cache in the same VRAM.
Ollama or LocalAI?
Ollama if you want a local chat and embeddings endpoint: ollama run, one binary, GPU detection and layer offload handled for you, and an API on port 11434. LocalAI if you need the wider OpenAI surface behind one endpoint, meaning transcription, speech synthesis, image generation and reranking as well as chat, dispatched to backends like llama.cpp, whisper.cpp, diffusers and vLLM. The cost is configuration: a YAML file per model naming the backend and prompt template, plus image variants pinned to a specific stack, from about 280 MB for CPU to roughly 3.5 GB for CUDA 12.
Is Open WebUI open source?
Not since v0.6.6, released 19 April 2025. It ships under a modified BSD-3-Clause that adds a branding-protection clause, and the project's own documentation states it is not OSI-approved. You may not alter, remove or obscure Open WebUI branding unless you have 50 or fewer end users in a rolling 30-day period, written permission as a contributor, or an enterprise licence. Code up to and including v0.6.5 remains plain BSD-3. For a household or small team the exemption covers you and nothing changes.
Why does SearXNG return no Google results?
Because Google is blocking your server, not because SearXNG is broken. It has no crawler of its own; it scrapes other engines, and those engines score hosting address ranges as automation regardless of how polite your headers or how slow your pace. A fresh instance on a datacenter VPS typically gets a too-many-requests exception from Google within a handful of searches, while DuckDuckGo and smaller engines keep working. A residential IP behaves far better, which inverts the usual self-hosting advice. Curate the engine list and accept a thinner mix.
Published . Last reviewed . Found something out of date? Tell us and we will fix it and log the change.