GPUs, transcoding and local AI: what hardware you actually need

"I want a GPU for my server" is two completely different purchases wearing one sentence. Video transcoding is a fixed-function encoder problem solved by a 6 W integrated graphics block. Local language model inference is a memory capacity problem solved by an expensive card with a lot of VRAM. The hardware that is excellent at one is usually mediocre or useless at the other.

Transcoding: Intel Quick Sync, and then stop reading#

Start by noticing how little transcoding you may need. Direct play, where the client can decode the file as it is, costs the server almost nothing: Jellyfin idles at 280 MB and does essentially no work streaming a file a TV can already play. Transcoding happens when the client cannot handle the codec, the container, the bitrate or the subtitle format. Fix the library and the client first and you may never need the encoder at all.

When you do need it, an Intel iGPU from an N100, N150, i3-12100, i5-12400 or anything newer is the answer. Jellyfin's own hardware guidance recommends Intel on any non-Apple hardware, names the N100 and the i5-11400 as the baseline recommendations, and explicitly removed 7th to 10th generation parts from that list because Intel deprecated their toolkit. AMD is ranked last there, for sub-par H.264 encoder quality. NVIDIA's RTX 50 series has the best encoder quality available, and it is also a 250 W card you do not need in a media server.

The power difference is the point. An N150 has a 6 W processor base power for the entire chip. A transcode session on an iGPU adds roughly 5 to 15 W over idle depending on generation and stream count; a discrete card idles at 8 to 20 W before it does any work at all. Over a year that gap is real money, as What a home server costs to run works through.

QSV, NVENC and VAAPI are not three tiers#

They are two hardware blocks and one API.

On an Intel chip, QSV and VAAPI drive the same silicon. QSV usually gives better HDR tone mapping and more format coverage; VAAPI is sometimes more reliable on older kernels. Try QSV first and fall back to VAAPI if a specific transform misbehaves. On AMD, VAAPI is the only path.

The device passthrough, per app#

The device node is /dev/dri, and passing it is only half the job. The container user also needs the host's render group, which is where most "hardware transcoding does not work" threads end.

getent group render   # note the GID, often 104 or 993

services:
  jellyfin:
    image: jellyfin/jellyfin
    devices:
      - /dev/dri:/dev/dri
    group_add:
      - "993"          # the render GID from the host
    volumes:
      - ./config:/config
      - ./cache:/cache
      - /srv/media:/media:ro

The per-app differences matter more than the similarities:

• Jellyfin: --device /dev/dri:/dev/dri plus group_add with the host render GID. Without the group, ffmpeg cannot open the device and silently falls back to software. Verify in the playback info overlay, which states whether the session is hardware accelerated. NVIDIA instead needs the NVIDIA Container Toolkit, the nvidia runtime and NVIDIA_DRIVER_CAPABILITIES=compute,video,utility.
• Emby: same device, different group mechanism. Emby's entrypoint reads a GIDLIST environment variable rather than Compose's group_add, so a Jellyfin snippet copied across will fail. And hardware transcoding requires an active Emby Premiere licence: the options appear in the UI regardless and the transcode still runs on CPU.
• Plex Media Server: --device /dev/dri:/dev/dri for Quick Sync, plus the NVIDIA toolkit for NVENC, plus the toggle in Settings, Transcoder, plus an active Plex Pass. Hardware transcoding has always been a paid feature, so the free tier is a CPU-only server. Run it with network_mode: host unless you want to fight discovery.
• Frigate: pass /dev/dri for OpenVINO detection and hardware decode. Two other settings decide whether the install survives: raise shm_size from Docker's 64 MB default using the documented per-camera formula (about 67 MB for one 1280x720 detect stream, 512 MB as a sane starting point), and point the detect role at a low-resolution substream rather than the 4K main stream. Getting the substream right often turns 90 percent CPU into 15.
• Immich: has two separate acceleration questions, video transcoding and machine learning. Both are x86-64 only for the good backends: CUDA, OpenVINO and ROCm do not exist on ARM, where you get ARM NN on Mali or RKNN on Rockchip and nothing else.

Local AI: VRAM is the binding constraint#

Everything about local inference follows from one fact: the model has to fit in the memory the compute unit can reach, and if it does not, performance falls off a cliff rather than a slope.

Ollama loads as many model layers into VRAM as fit and runs the remainder on the CPU. Missing the fit by two layers does not cost you a small percentage. Throughput typically drops several times over, because every token now has to cross the PCIe bus. Watch the layer count in the server log; if it is not offloading all of them, drop to a smaller quantization or a shorter context rather than accepting the split.

Orders of magnitude worth memorizing:

Those CPU figures scale with memory bandwidth, not core count. This is the single most misunderstood thing about local inference: a 16-core server with dual-channel DDR4 is barely faster than an 8-core one, so buying cores to speed up a local model is money set on fire. The only change that matters is a GPU with enough VRAM to hold the whole model, or Apple Silicon, where unified memory means a 32 GB Mac can run models a 12 GB discrete card cannot.

Quantization is the lever you actually have. The default tags in Ollama's library are typically Q4_K_M, a four-bit compromise at roughly half the memory of Q8. Pulling :latest gets you the compromise rather than the model as published. If you have VRAM headroom, Q6 or Q8 are noticeably better on reasoning and code. Q3 exists and degrades badly.

Three defaults will surprise you. Context length defaults to 4096 tokens regardless of what the model advertises, so a 20,000 token document is silently truncated from the front. Models unload after 5 minutes idle, so the next request pays the full load time again. And OLLAMA_MAX_LOADED_MODELS defaults to 3, so three different models can quietly fill a GPU you thought was free.

# a 128k-capable model still runs at 4096 unless you say otherwise
OLLAMA_CONTEXT_LENGTH=32768 OLLAMA_KEEP_ALIVE=-1 ollama serve

Why one GPU rarely covers both#

The two purchases pull in opposite directions. A transcoding GPU should be small, cheap, low-idle and have a modern media engine; VRAM is irrelevant. An inference GPU should have as much VRAM as you can afford; its media engine is irrelevant. The overlap, an RTX 3060 12 GB, is a compromise on both and adds meaningful idle draw to a machine that is idle 95 percent of the time.

The sane setup for most people: an Intel iGPU in the always-on server for transcoding and detection, and, if you genuinely want local models, either a second machine you power on when you use it, or Apple hardware you already own. Running Open WebUI against a model on a desktop that is awake anyway costs nothing extra.

What Immich and Frigate actually want#

Immich wants RAM before it wants a GPU. The docs ask for 6 GB minimum and 8 GB recommended, and the immich-machine-learning container holding CLIP and face models resident is most of that. Raising MACHINE_LEARNING_WORKERS loads a second full copy of the models, which roughly doubles resident memory, and the failure mode is an exit code 137 that looks like broken search rather than an out-of-memory kill. GPU acceleration for ML is a nice-to-have that shortens the initial import; the initial import will peg every core for hours either way.

Frigate wants an Intel iGPU, and specifically not a big GPU. Decoding costs more than detecting, and only the GPU path accelerates decode. The documentation no longer recommends a Coral for new installs outside low-power builds, because OpenVINO on a modern iGPU runs inference in roughly 4 to 15 ms against the Coral's 10 ms and does the decode as well. Version 0.17 also dropped support for NVIDIA GTX 900 series cards, which is another data point on old NVIDIA hardware in this role. Plan the disk before the detector: at roughly 43 GB per camera per day for a 4 Mbit/s continuous stream, six cameras is about 1.8 TB a week.

What to do next#

If you are still choosing a machine, Choosing home server hardware puts the iGPU in the right place in the buying order (second, after RAM). If the media stack is the point, Jellyfin vs Plex settles which server you are passing the device into. And if local models are the real goal, read the Ollama profile and Ollama vs LocalAI before you buy a card, because the VRAM number decides the outcome and nothing else comes close.