Migration
Replace ChatGPT: run a chat model on your own hardware
Ollama plus Open WebUI gives you a private chat interface with no per-token cost. It also gives you a model that is measurably worse than the one you are paying for, and that gap is the whole decision.
Can you replace ChatGPT with a self-hosted model?
You can replace the interface and most of the everyday work with Ollama serving a quantized model behind Open WebUI, on hardware you already own. What you lose is capability: an 8-billion-parameter model at four-bit quantization is roughly 5 GB and handles summarizing, rewriting, extraction and questions over your own documents, but it fails on multi-step reasoning and long multi-file code where a hosted flagship succeeds. VRAM is the binding constraint, and with no GPU at all you are looking at 5 to 15 tokens per second, which is batch work rather than conversation. Run both for a month before you cancel anything.
The model you run at home is worse than the model you rent. Not slightly, and not in a way that a better system prompt fixes. Everything below is written on that assumption, because a page that pretends otherwise wastes an afternoon of your life and a few hundred dollars of graphics card.
What you are actually giving up#
Capability, measured in parameters and bits.
Quantized model files follow a usable rule: about 0.6 GB per billion parameters at four-bit quantization. An 8B model is 4 to 5 GB, a 30B-class model is around 18 GB, a 70B is around 40 GB. That arithmetic is the entire hardware conversation, because the model has to fit in memory the accelerator can reach.
Here is what each tier honestly does.
An 8B at Q4_K_M summarizes, rewrites, extracts structured fields from messy text, classifies, drafts an email, explains a stack trace and answers questions about documents you hand it. It is genuinely useful and it is fast. It fails at anything requiring several correct steps in a row: arithmetic with carries, a proof, a refactor across files, a fact it has not memorized. The failure mode is not "I do not know". It is a fluent, confident, wrong answer.
A 30B-class model at Q4 is noticeably better at instruction-following and code, holds a longer thread of reasoning, and hallucinates less. It needs roughly 18 GB, so a 24 GB card. It is still not a current hosted flagship, and you will notice the moment you give it a hard problem.
A 70B at Q4 is around 40 GB and drops to roughly 1 to 3 tokens per second on CPU, which is not usable interactively. Making it fast means two 24 GB cards, one 48 GB card, or a large-memory Apple Silicon machine.
Quantization itself is a cost people forget. The default tags in the Ollama library are typically Q4_K_M, the compromise build rather than the model as published. Pulling :latest costs you a measurable amount on reasoning and code against a Q6 or Q8 tag, so check the tag list if you have the VRAM.
VRAM is the binding constraint, not cores. Ollama loads as many layers into VRAM as fit and runs the rest on the CPU, so performance is a cliff and not a slope: a model that misses the fit by two layers is several times slower, because every token now crosses the PCIe bus. CPU-only inference is memory-bandwidth-bound, so a 16-core server on dual-channel DDR4 is barely faster than an 8-core one. Expect 5 to 15 tokens per second for an 8B on a modern desktop CPU. That is slower than you read.
You also give up the polish: maintained mobile apps, voice mode, a browsing tool that works, and somebody else swapping in a better model while you sleep.
The case that survives all of that#
Three things do not degrade. Nothing leaves the machine, so client documents, medical letters and half-finished ideas stay on your disk. There is no per-token cost, so summarizing 4,000 emails is a question of patience rather than budget. And there is no rate limit and no policy change, so a script that hammers localhost:11434 overnight simply works. For retrieval over your own documents a small model is often good enough anyway, because the hard part is finding the right paragraph and the model only has to read it back.
Get your conversations out first#
Do this before you cancel anything. In ChatGPT: Settings, Data controls, Export data. You get an email with a link to a zip containing conversations.json plus an HTML viewer, and the link expires, so download it the same day.
Nothing imports that file into Open WebUI, so treat it as an archive rather than a migration. Mine it anyway, because it tells you what you actually use the thing for:
unzip -o chatgpt-export.zip -d chatgpt-export
jq -r '.[].title' chatgpt-export/conversations.json | sort | uniq -c | sort -rn | head -30Pick five real prompts out of that list. They are your acceptance test, and they are more honest than any benchmark.
Ollama
Pull and run quantized language models locally with one command and an HTTP API
150 MB idle · ops load 2/5 · MITLocal AIOpen WebUI
Chat front end for local and remote models, no longer under a plain BSD licence
600 MB idle · ops load 2/5 · Open WebUI License (BSD-3-Clause plus a branding clause)Local AILocalAI
One OpenAI-compatible API in front of many inference backends, not just text
300 MB idle · ops load 3/5 · MITWhich one to run#
For almost everyone: Ollama for inference, Open WebUI for the interface. Ollama is one binary and one blob directory with no database. Open WebUI is one container and one volume, and it can front a local Ollama and a hosted OpenAI-compatible endpoint at the same time under one conversation history, which is the setup that survives contact with a hard question.
| Your situation | Run | Why |
|---|---|---|
| 12 to 24 GB GPU, want chat and document search | Ollama plus Open WebUI | One command, one container, GPU detection handled |
| Apple Silicon with 32 GB or more unified memory | Ollama natively, not in Docker | Unified memory holds models a 12 GB discrete card cannot |
| No GPU, but you want overnight batch summarizing | Ollama alone, scripted against the API | 8 tokens per second is fine when nobody is waiting |
| You also need transcription, TTS or image generation | LocalAI | One OpenAI-compatible endpoint across many backends |
| Household of five who each want their own history | Open WebUI with accounts and group permissions | Per-user chats, admin control over which models each group sees |
Ollama vs LocalAI goes deeper. The short version: LocalAI's breadth costs you a YAML file per model and an image variant pinned to your exact CUDA, ROCm or oneAPI version.
What it actually costs#
Hardware, mostly, and it is a single number.
| VRAM available | Largest model that fits well | What it feels like |
|---|---|---|
| None, CPU only | 3B to 8B at Q4 | 5 to 15 tokens per second, batch work |
| 8 GB | 8B at Q4, short context | Quick drafting, tight on context |
| 12 GB | 8B at Q6 or Q8, or a 14B at Q4 | The first tier that stops feeling like a compromise |
| 16 GB | 14B at Q4 comfortably | Good general assistant |
| 24 GB | 30B class at Q4, about 18 GB | Real code help, room for a long context |
| 48 GB or more | 70B at Q4, about 40 GB | Closest you get, at serious cost and power draw |
Two adjustments to that table. The file size is a floor, not a total: raising num_ctx allocates a proportionally larger KV cache in VRAM, so budget another 1 to 2 GB before you decide something fits. And OLLAMA_MAX_LOADED_MODELS defaults to 3, so three models you tried once can quietly occupy the card.
Disk is sneakier. A casual ollama pull habit fills a 256 GB disk inside a month, so set OLLAMA_MODELS to a large disk before you pull anything. Do not back the blobs up: every published model is re-pullable, so the real backup is ollama list in a text file plus any custom Modelfile and any GGUF you imported and no longer have. See Backups that actually restore, GPUs, transcoding and local AI for card choice, and What a home server costs to run for what an idling GPU adds to the bill.
What breaks and how to tell early#
The moment you ask it something hard. This arrives in week one. Run those five exported prompts and read the answers properly instead of skimming them for fluency. If three of the five are worse in a way that matters, you learned it cheaply.
An empty model list in Open WebUI. OLLAMA_BASE_URL defaults to http://localhost:11434, which inside Docker is the Open WebUI container, not your host. No error, just no models. Put both on one compose network and use the service name.
A setting that will not change. Open WebUI copies config variables into its database on first launch and reads them from there, so editing compose and restarting does nothing. Change it in the admin UI, or set ENABLE_PERSISTENT_CONFIG=False.
A restore that logs everyone out. Set WEBUI_SECRET_KEY explicitly. If you do not, one is generated into the data volume, and restoring without it invalidates every session and makes stored credentials undecryptable.
Silent storage growth. Deleting a chat does not delete its attached files or their embeddings, which is a disk problem and a privacy one.
The point of no return#
There is not one for the software. Both uninstall cleanly and neither owns data you cannot leave with.
The point of no return is deleting your ChatGPT account, because the export link expires and the history does not come back. Sequence it: export, stand up the local stack, run both in parallel for a full month with the subscription still active, and note every time you reached for the hosted model instead. Short note, cancel. Long note, you lost a weekend and learned something, which is what When not to self-host is for.
What to do next#
Pull an 8B model, point Open WebUI at it and run your five prompts today. Read GPUs, transcoding and local AI before buying a card, size the box with Stack planner, and if search is the other thing you want off a vendor's servers, Replacing Google Search is the companion migration. The rest of the category is in Local AI.
Questions#
How much VRAM do I need to replace ChatGPT locally?
Twelve gigabytes is the point where this stops feeling like a compromise for chat and drafting, and 24 GB is where a 30B-class model at four bits fits with room for context. The file size rule is about 0.6 GB per billion parameters at Q4: an 8B model is 4 to 5 GB, a 30B is around 18 GB, a 70B is around 40 GB. Add 1 to 2 GB on top for the KV cache once you raise the context length. Under 8 GB with no GPU you are running 1 to 3B models that are not good enough for real work.
Is a local model good enough for coding?
For autocomplete, boilerplate in a popular language, writing a regex, explaining a stack trace and small single-file edits, yes. For a change that spans six files and needs the model to hold the architecture in its head, no, and a 30B at four bits does not close that gap either. The realistic pattern is a local model for the fast and private things and a hosted API for the hard ones, which is exactly what Open WebUI's multi-backend model list is for.
Why is my local model ignoring half of a long document?
Ollama's OLLAMA_CONTEXT_LENGTH defaults to 4096 tokens no matter what the model advertises. Feed in a 20,000 token document and the front of it is silently dropped, with no error and no warning, so you get an answer that confidently discusses the last quarter of the file. Raise it per request with num_ctx or globally with the environment variable, and remember a bigger context allocates a proportionally bigger KV cache in VRAM.
Is Open WebUI still open source?
Not in the OSI sense, since v0.6.6 on 19 April 2025. It ships under a modified BSD-3-Clause with a branding-protection clause, and the project's own documentation says plainly that this is not an OSI-approved licence. You may not alter or remove Open WebUI branding unless you have 50 or fewer end users in a rolling 30-day period, written permission as a contributor, or an enterprise licence. For a household that exemption covers you completely. For a white-labeled portal at a 200-person company it does not.
Should I use Ollama or LocalAI?
Ollama if you want chat and embeddings, because it is one command and one port. LocalAI if you need transcription, speech synthesis, image generation or reranking behind the same OpenAI-compatible endpoint, and you accept a YAML file per model plus an image variant pinned to your exact accelerator. The CPU LocalAI image is around 280 MB compressed; the CUDA 12 variant is roughly 3.5 GB and the Intel GPU build is over 4 GB.
Is it safe to expose Ollama on my network?
The API has no authentication of any kind. Setting OLLAMA_HOST=0.0.0.0 publishes an endpoint that will pull models, run inference and delete models for anyone who can reach port 11434, and internet-wide scans for open 11434 have been running for a while. Keep it on localhost, let Open WebUI reach it over the Docker network, and if it must travel further put it behind a token-checking reverse proxy or bind it to a WireGuard interface.
Sources#
- Ollama docs, FAQ: model paths, OLLAMA_HOST, keep-alive, context, concurrency
- Ollama docs, GPU support and compute requirements
- Ollama README and API endpoint
- Open WebUI docs, licence terms and the branding clause
- Open WebUI LICENSE file
- Open WebUI docs, environment variable configuration and persistent config
- Open WebUI docs, quick start, image tags and architectures
- LocalAI docs, container images and variants
- LocalAI repository, licence, backends and quick start
Published . Last reviewed . Found something out of date? Tell us and we will fix it and log the change.