Running Local LLM Inference in Docker on Apple Silicon: The GPU Passthrough Problem (2026)

Every team that gets local inference working on a Mac eventually tries to containerize it, and most of them hit the same wall within an hour. The model runs beautifully in a native process, MLX or llama.cpp driving the Apple GPU at full speed, and then it goes into a Dockerfile and the throughput collapses. Nothing in the logs explains it. The container starts, the model loads, generation works, it is just slow in a way that feels broken. The cause is not your code. Docker on Apple Silicon cannot give a container access to the GPU, so everything inside the container runs on CPU, and for a workload that lives and dies on GPU throughput that is the difference between a usable service and a demo you quietly abandon. Understanding why this happens, and what to do instead, is the difference between shipping local inference on Mac hardware and giving up on it.

Where the Boundary Actually Sits

Docker does not run containers natively on macOS. There is no such thing as a macOS container in the way there is a Linux container, so Docker Desktop spins up a small Linux virtual machine, using Apple's Virtualization framework, and your containers run as Linux processes inside that guest. This is invisible most of the time and it is exactly why Docker on a Mac feels the same as Docker on Linux. The problem shows up only for workloads that need hardware the virtual machine does not have.

The Apple GPU is one of those. When the Virtualization framework boots the Linux guest, it presents a virtual machine with CPU and memory, but it does not expose the Metal capable GPU to the guest operating system. There is no virtual GPU device, no passthrough path, and no driver inside the Linux guest that could talk to the Apple GPU even if the device were present, because Metal is a macOS framework with no Linux implementation. On a Linux server with an NVIDIA card, the NVIDIA container toolkit maps the physical device into the container and the CUDA libraries inside the container talk to it. On a Mac there is no equivalent because there is nothing to map: the guest cannot see the GPU, so the container two layers up certainly cannot. This is a property of the platform, not a gap someone will patch next quarter.

The practical consequence is that any inference runner you put in a Linux container on a Mac falls back to its CPU backend. llama.cpp compiles a CPU path and uses it. MLX does not even enter the picture, because MLX is built on Metal and does not run on Linux at all, so an MLX based service cannot be containerized on a Mac even in principle. We compared those two runners on their native fast paths in the Metal backend versus MLX writeup, and the containerized case erases that comparison entirely because neither one gets the GPU.

What Containerization Costs, Measured

These are directional figures from a 128GB M5 Max running a 14B class model at 4-bit quantization, batch size one, a short prompt so prefill is not the story, comparing a native MLX process against the same class of model running in llama.cpp inside a Docker container that has fallen back to CPU. They are framed to show the shape of the gap, not as universal constants, because the exact numbers shift with model size, quantization, and thread count.

Setup	GPU used	Decode throughput (tok/s)	First token (s)	Notes
Native MLX process	Yes, Metal	88	0.4	Full GPU, unified memory
Native llama.cpp, Metal	Yes, Metal	79	0.5	Full GPU
llama.cpp in Docker, CPU only	No	9	2.1	Container fallback
llama.cpp in Docker, CPU, more threads	No	12	1.9	Diminishing returns

Read the GPU column first, because it is the whole explanation. The two native setups use Metal and land in the same performance neighborhood. The two containerized setups do not touch the GPU and land roughly an order of magnitude lower, and no amount of thread tuning closes that gap, because the CPU cores on an M5 Max, generous as they are, are not the part of the chip built for the dense matrix work that decode is. Throwing more container threads at it moves throughput from nine to twelve tokens per second and then stalls, which tells you the ceiling is the hardware path, not the parallelism. For an interactive service this is the difference between text that streams faster than a person reads and text that limps out slower than they type.

The Deployment Pattern That Actually Works

The fix is to stop trying to put the GPU workload in the container and instead run the model natively while keeping everything around it containerized. The model server runs as a native macOS process, MLX or llama.cpp with Metal, bound to a local port and exposing an OpenAI compatible endpoint. Your application, your API layer, your queue workers, whatever else the service needs, all of that goes in containers as usual and talks to the model over that local HTTP endpoint. The container reaches the native server through host.docker.internal, which resolves from inside the Linux guest to the Mac host, so your application code does not care that the model lives outside the container.

# docker-compose.yml, app side only
services:
  api:
    build: .
    environment:
      # the model runs natively on the host, not in a container
      LLM_BASE_URL: "http://host.docker.internal:8080/v1"
    ports:
      - "3000:3000"

# on the host, native, with GPU access
mlx_lm.server --model mlx-community/your-model-4bit --port 8080

This keeps the GPU workload on the only path that can reach the GPU while letting the rest of the stack stay in the container world where it belongs. The model server is the one native piece, managed as a launchd service or a simple supervised process so it restarts cleanly, and it is also the piece you would run as a small pool if you are serving more than one model, which we broke down in the concurrent multi model serving post. Choosing between the two server implementations for that native layer is its own decision, and the tradeoffs between them are covered in the mlx-lm server versus llama-server comparison.

The one thing this pattern gives up is the fully self contained image. You cannot hand someone a single container that includes the model and runs fast on a Mac, because the fast path is outside the container by definition. What you can hand them is a compose file plus a one line native server command, which is a small operational cost for keeping the GPU. If your deployment target is genuinely a Linux server with a real GPU, none of this applies and you should build the normal all in one image; the native split is specifically the Apple Silicon accommodation.

When to Just Use Linux Instead

There is an honest version of this decision where the answer is to not run inference on the Mac at all. If your production target is a cloud Linux box with an NVIDIA card, then developing against a native Mac process and deploying to a CUDA container is two different runtimes and two different performance profiles, and you are better off developing against the same Linux GPU stack you deploy to. Apple Silicon inference earns its place when the Mac itself is the deployment target: an on premises box, a workstation running a local feature, a privacy sensitive workload that has to stay on the machine, an edge unit where a Mac mini or Studio is the hardware. In those cases the native split above is the right shape. When the Mac is only a development convenience for a Linux production target, the container mismatch is a reason to move development onto Linux hardware, not a problem to engineer around.

When This Applies to Your Stack

If you are deploying local inference on Apple Silicon and you assumed Docker would treat the Apple GPU the way it treats an NVIDIA card, that assumption will cost you an order of magnitude in throughput the moment you containerize the model. There is no passthrough to configure, because the GPU is never exposed to the Linux guest the containers run in, and MLX does not run in Linux containers at all. The pattern that keeps you fast is to run the model server as a native macOS process with Metal and containerize everything else, connecting to the model over a local endpoint. Reserve the fully containerized approach for actual Linux GPU targets. Measure decode throughput both ways before you commit to a container image, because the native versus container gap is not a tuning problem you will close later, it is the platform telling you where the GPU boundary is.

If your team is standing up local inference on Mac hardware and the deployment story is fighting you, the native server plus containerized application split is usually the unlock, and getting that boundary right, along with the process supervision and model pooling around it, is the kind of infrastructure work we do at Contra Collective when we help teams take local inference from something that runs on one laptop to something that runs as a real service.

FAQ

Can I pass the Apple GPU into a Docker container on a Mac? No. Docker Desktop runs containers inside a Linux virtual machine created by Apple's Virtualization framework, and that virtual machine is never given access to the Apple GPU. There is no device to pass through and no Metal driver inside the Linux guest, so the container has no path to the GPU regardless of configuration.

Why is my model so much slower in Docker than running it directly? Because inside the container it is running on CPU. The GPU accelerated path is unavailable in the container, so llama.cpp falls back to its CPU backend and MLX cannot run at all. CPU decode on an M5 Max is roughly an order of magnitude slower than the Metal path for a typical model, which is the slowdown you are seeing.

Does this affect llama.cpp and MLX the same way? Not quite. llama.cpp will run in a Linux container but only on CPU, so it works and is slow. MLX is built on Metal and has no Linux build, so it cannot run in a Mac Linux container at all. Both lose the GPU; MLX loses the ability to run entirely.

What is the recommended way to deploy local inference on a Mac then? Run the model server as a native macOS process with Metal, exposing a local HTTP endpoint, and put your application and supporting services in containers that reach the model through host.docker.internal. This keeps the GPU workload on the only path that can use the GPU while keeping the rest of the stack containerized.

Will Docker or Apple add GPU passthrough later? Treat it as not available and design around it. The limitation comes from the virtualization layer not exposing the GPU and from Metal being macOS only, which is a structural boundary rather than a missing feature, so the safe engineering assumption is that the native split is the pattern for the foreseeable future.