LocalAI presents an OpenAI-compatible REST API and routes each request to whichever backend serves that model, covering text generation, embeddings, speech to text, text to speech, image generation and reranking. It replaces the OpenAI API for applications you do not want to rewrite, running the work on your own hardware.
Our verdict on LocalAI#
The differentiator is breadth behind one API contract. Ollama gives you text and embeddings through its own API plus an OpenAI-compatible shim; LocalAI aims at the whole OpenAI surface, chat, embeddings, audio transcription, speech synthesis, image generation and reranking, and dispatches each to an appropriate backend from a set that now numbers in the dozens including llama.cpp, whisper.cpp, diffusers, vLLM and MLX. That breadth is the cost. Where Ollama's mental model is ollama run, LocalAI's is a YAML file per model naming the backend, the template and the parameters, and a backend gallery you install from. Image size follows: the plain CPU image is around 280 MB compressed, but the CUDA 12 variant is roughly 3.5 GB and the Intel GPU one over 4 GB, and you must pick the variant that matches your accelerator and driver. Choose LocalAI when you need more than chat behind one endpoint. Choose Ollama when you do not.
Who LocalAI is for#
Choose it if
- Applications already written against the OpenAI SDK where changing the base URL is the entire migration.
- Anyone needing transcription, speech synthesis or image generation alongside chat from a single service.
- Operators who want per-model control over prompt templates, context size and backend rather than accepting a packaged default.
- Air-gapped or restricted networks, where one service and a directory of model files is easier to justify than five.
Look elsewhere if
- All you want is a local chat model, where Ollama is one command and LocalAI is a YAML file.
- You are bandwidth or disk constrained, since the GPU image variants run several gigabytes before any weights.
- You need a single supported path, because the backend and accelerator matrix means your exact combination may be less travelled than the defaults.
What running it actually looks like#
One container on port 8080, but choosing the tag is the real decision. There is a plain CPU image, NVIDIA CUDA 12 and 13 variants, AMD ROCm, Intel oneAPI, Vulkan, and Jetson L4T builds for arm64. Compressed sizes differ by an order of magnitude: around 280 MB for the CPU image and roughly 260 MB on arm64, against about 3.5 GB for CUDA 12 and over 4 GB for the Intel GPU build. Mount persistent volumes for /models, /backends, /configuration and /data, because backends are installed at runtime from a gallery rather than baked into the image, and losing that volume means re-downloading them. Standard images ship no preconfigured models. GPU variants need the matching host driver and the container toolkit. First useful start involves pulling a model and, usually, writing or importing its YAML.
Resource profile#
- Memory
- 300 MB idle, around 6500 MB in ordinary use. The server is small until a model loads. Typical here is one 7 to 8 billion parameter model at four-bit quantization; running text, embeddings and speech models simultaneously means adding all of them, because LocalAI will happily keep several backends resident.
- CPU and acceleration
- CPU inference is memory-bandwidth-bound like any llama.cpp-derived runtime. Diffusion image generation on CPU is the outlier: it will saturate every core for minutes per image and is not worth doing without a GPU.
- Storage growth
- Models dominate, and backends now add to it: each backend you install from the gallery unpacks into the backends directory, and a CUDA-capable Python backend can be gigabytes on its own.
- Operational load
- 3 of 5, Moderate. One container, but a real matrix: a per-model YAML config, a backend gallery whose installs must match your accelerator, and image variants tied to a specific CUDA, ROCm or oneAPI version. Upgrades usually work; a driver or CUDA mismatch turns into an afternoon.
Figures describe a small single-household install and are the working assumptions behind our stack planner. Your numbers will differ with library size, user count and hardware. See methodology.
Data and backup surface#
Four directories and no database. /models holds the weight files and the per-model YAML that defines the backend, prompt template, context length and parameters. /configuration holds instance-level config. /backends holds the runtimes installed from the gallery. /data holds generated output and uploads. A live file copy is safe since nothing is a transactional store, though copying while a model is downloading gets you a partial file. The only genuinely irreplaceable part is the YAML: model weights are re-downloadable and installed backends are re-installable, so a small backup of /models/*.yaml plus /configuration covers the work you actually did. Exclude /backends and the weight files unless your restore has to work offline.
Derived backup shape
File copy. Copy the config and data directories. Safe while running for anything that writes atomically.
Traps and surprises#
The image variant must match your accelerator
Tags are pinned to a specific stack: CUDA 12, CUDA 13, ROCm, Intel oneAPI, Vulkan or plain CPU. Run the CUDA 12 image against a driver that only supports an older runtime and it starts, loads a model and silently falls back to CPU, so the only symptom is that it is slow. Check the startup log for which backend actually initialized rather than trusting that GPU passthrough worked.
Backends are installed at runtime, not shipped in the image
Recent LocalAI moved backends out of the container into an installable gallery, which is why the base image is only a few hundred megabytes. If you do not persist the backends directory, every container recreation re-downloads them, and on an air-gapped host the model simply fails to load with a missing-backend error. Mount /backends as a volume before you do anything else.
Per-model YAML is where the behavior lives
A model that produces rambling or role-confused output is usually a prompt template mismatch in its YAML, not a bad model. LocalAI does not infer the chat template from the GGUF metadata in every case, so a Llama-format template on a Mistral-format model gives plausible-looking garbage. Start from the gallery entry for the model rather than hand-writing the config, and compare against the model card's template.
API keys are the only access control
LocalAI has no user accounts. If you do not configure the API key list, the endpoint is fully open, and it will load and run any model in your models directory for anyone who can reach port 8080. Set the keys, keep the service off the public internet, and remember that keys are static strings in config with no rotation or per-key scoping beyond usage attribution.
Concurrent models silently exhaust memory
Because one endpoint fronts many models, an application that calls chat, then embeddings, then transcription can end up with three backends resident at once. There is no global memory budget that stops this, so the third load either triggers the OOM killer or spills into swap and takes the whole box down with it. Constrain which models are loadable, and set container memory limits so the failure is contained.
Specifications#
- Category
- Local AI and private search
- Licence
- MIT (Permissive)
- Written in
- Go / C++ / Python
- First release
- 2023
- Maturity
- Stable
- Datastore
- None; model files and per-model YAML on disk
- Default ports
8080- Ships as
- docker, native binary, helm chart, kubernetes
- arm64 builds
- Yes
- Identity
- Not applicable. No user accounts; access control is a list of static API keys checked against the Authorization header, so anything richer belongs in a reverse proxy.
- Replaces
- OpenAI API, AssemblyAI, ElevenLabs, Stability AI API
- Project site
- localai.io
- Source
- github.com/mudler/LocalAI
- Documentation
- localai.io/docs
Alternatives to LocalAI#
Everything else in local ai and private search, closest in operational weight first.
| Tool | Ops load | Idle RAM | Licence |
|---|---|---|---|
| Ollama | 2, Light | 150 MB | MIT |
| Open WebUI | 2, Light | 600 MB | Open WebUI License (BSD-3-Clause plus a branding clause) |
| SearXNG | 2, Light | 150 MB | AGPL-3.0-or-later |
Where LocalAI comes up elsewhere#
Head to head
Migration guides
Sources#
Facts on this page were checked against the project's own material. Where a number is our own estimate rather than a published figure, the text says so.
Published . Last reviewed . Found something out of date? Tell us and we will fix it and log the change.