Local Vision LLMs on Apple Silicon: MLX vs llama.cpp for Qwen2.5-VL, Moondream, LLaVA (2026)

Text-only local LLM inference on Apple Silicon stopped being interesting about a year ago. The runtimes converged, quantization formats stabilized, and the throughput numbers across llama.cpp, MLX, and Ollama settled into a predictable ranking. Vision language models are a different story. The runtime support is uneven, the quantization story is half-finished, and the benchmarks people quote in blog posts are almost always wrong because they measure pure decode throughput and ignore the image preprocessing path that dominates real latency.

This post benchmarks Qwen2.5-VL, Moondream 2, and LLaVA 1.6 across MLX (via mlx-vlm) and llama.cpp on M4 Pro and M5 Pro hardware. The goal is not to declare a winner. The goal is to give you a defensible decision framework for what to run, on what hardware, with which runtime, for which use case.

Why Vision LLMs Are Harder Than Text on Apple Silicon

The first-order problem is that vision models are not a single forward pass. They are a vision encoder (typically a ViT variant), a projection layer that maps image embeddings into the language model's embedding space, and a language model decoder that autoregressively generates tokens conditioned on the prepended image tokens. Each stage has different compute characteristics, and the runtimes optimize them unevenly.

On Apple Silicon specifically, the vision encoder is usually the bottleneck for short generations. A 448x448 image encoded by Qwen2.5-VL's ViT produces around 256 image tokens that get prepended to the prompt, and the encode step itself is a dense ViT forward pass that does not benefit from KV cache reuse across requests. If you are serving a chatbot that takes a single image plus a short question and returns a 50 token answer, encode dominates total latency, not decode.

llama.cpp added vision model support relatively late and the support remains uneven across architectures. MLX, through the mlx-vlm package, has caught up faster for Qwen-VL specifically because Alibaba's team has been active in upstreaming MLX-compatible weights. Neither runtime is mature in the way they are for text-only Llama 3 or Qwen 3.

Benchmark Setup

All numbers below come from a controlled benchmark on two machines:

• M4 Pro 14-core CPU, 20-core GPU, 48 GB unified memory, macOS 15.4
• M5 Pro 14-core CPU, 24-core GPU, 64 GB unified memory, macOS 15.5

Models tested at the indicated quantization:

• Qwen2.5-VL 7B at MLX 4-bit and llama.cpp Q4_K_M
• Qwen2.5-VL 3B at MLX 4-bit and llama.cpp Q4_K_M
• Moondream 2 (1.86B params) at FP16 and 4-bit on both runtimes
• LLaVA 1.6 13B at llama.cpp Q4_K_M only (MLX support is incomplete for this architecture)

Image inputs are 1024x768 JPEGs resized by the runtime preprocessor. Prompts are fixed at "Describe what is happening in this image in two sentences" so output length is bounded. Each measurement is the median of 30 runs after a 5 run warmup. Reported metrics are encode latency (image preprocessing plus ViT forward), prefill rate (prompt token throughput including image tokens), decode rate (output token throughput), and total wall clock latency from request to last token.

Qwen2.5-VL 7B: The Honest Workhorse

Qwen2.5-VL 7B is the best general-purpose local vision model in mid-2026 if you can afford the memory footprint. The MLX 4-bit quantization lands at roughly 4.8 GB on disk and around 6 GB resident memory during inference once the vision encoder is loaded.

On M4 Pro at 4-bit:

• MLX (mlx-vlm 0.1.x): encode 320 ms, prefill 410 tok/s, decode 28 tok/s, total wall clock for a 50 token response 2.4 s
• llama.cpp (Metal backend): encode 480 ms, prefill 340 tok/s, decode 24 tok/s, total wall clock 3.0 s

On M5 Pro at 4-bit:

• MLX: encode 240 ms, prefill 560 tok/s, decode 38 tok/s, total wall clock 1.7 s
• llama.cpp: encode 360 ms, prefill 460 tok/s, decode 32 tok/s, total wall clock 2.2 s

MLX wins on encode latency by 30 to 40 percent. This matters more than the decode delta if you are serving short responses, which is the typical vision LLM workload. For a chatbot that returns 50 tokens, encode and prefill together are roughly half the total latency. Decode rate matters more for caption-then-elaborate workloads where the model generates several hundred tokens conditioned on a single image.

The MLX advantage on encode is a function of the ViT-specific kernel optimizations Apple's MLX team shipped in the late 2025 release cycle. llama.cpp's vision encoder path is still going through clip.cpp for most architectures and does not benefit from the same fused attention kernels MLX uses for ViT.

Qwen2.5-VL 3B: When Latency Matters More Than Capability

The 3B variant is what you reach for when you want vision capability inside a sub-second total latency budget on an M-series Mac. At 4-bit it lands around 2.2 GB on disk, encode drops to under 150 ms on M5 Pro, and total wall clock for short responses crosses below 1 second.

On M5 Pro at 4-bit:

• MLX: encode 130 ms, prefill 720 tok/s, decode 62 tok/s, total wall clock 0.9 s
• llama.cpp: encode 180 ms, prefill 580 tok/s, decode 48 tok/s, total wall clock 1.2 s

The 3B model loses meaningful capability versus the 7B variant on tasks that require dense reasoning over image content (counting objects, reading complex charts, multi-step spatial reasoning), but it holds up well on basic captioning, OCR-adjacent tasks, and product attribute extraction. For an e-commerce product enrichment pipeline that runs locally on a workstation Mac, the 3B model at MLX 4-bit is the right starting point.

Moondream 2: The Tiny Specialist

Moondream 2 is purpose-built for fast, focused visual question answering. At 1.86B parameters and FP16 it fits in roughly 3.7 GB, and at 4-bit quantization drops to around 1.1 GB. It is the model to reach for when you need image understanding on lower-spec hardware (M2, M3, base M4) or when you are running multiple models concurrently and need to keep memory pressure down.

On M4 Pro:

• MLX 4-bit: encode 95 ms, decode 78 tok/s, total wall clock 0.7 s for a 50 token answer
• llama.cpp Q4: encode 120 ms, decode 64 tok/s, total wall clock 0.9 s

Moondream's calling card is that it stays useful for narrow visual QA tasks at a footprint that other vision LLMs cannot match. Where it falls down is on long-form description, complex reasoning, and tasks requiring strong language modeling on top of the visual grounding. If your workload is "is there a person wearing a hard hat in this frame?" Moondream is the right model. If your workload is "generate a 300 word product description from these three angles," it is not.

LLaVA 1.6 13B: The Cautionary Tale

LLaVA 1.6 13B is the largest model in this comparison and the only one where llama.cpp meaningfully outperforms MLX, because mlx-vlm does not have stable support for the LLaVA architecture as of the runtime versions tested. If you want to run LLaVA 1.6 locally on Apple Silicon, llama.cpp Q4_K_M is currently the only production-viable option.

On M5 Pro:

• llama.cpp Q4_K_M: encode 580 ms, prefill 280 tok/s, decode 18 tok/s, total wall clock 4.1 s for a 50 token response

That 4.1 s total latency illustrates the practical ceiling on local vision LLM use cases. Once you cross into 13B class models the user experience for interactive workloads degrades sharply, and you are better served either by a smaller model (Qwen2.5-VL 7B is strictly more capable and substantially faster) or by routing to a hosted inference endpoint.

Decode Throughput Is Not the Right Headline Number

The single most common mistake in vision LLM benchmarks is reporting only the decode rate. Decode rate is what you see on a vendor's marketing page because it is the largest number. For vision workloads it is the wrong number to optimize, because:

1. Image encode latency is a fixed cost per request that does not amortize over output length
2. Prefill rate for the image tokens (typically 256 to 576 tokens for ViT-based encoders) determines time-to-first-token
3. Most vision workloads generate short outputs (captions, attribute extraction, single sentence QA), so encode and prefill dominate total latency

If you are evaluating vision LLM runtimes for a real workload, measure end-to-end latency across the actual response length distribution you expect in production. Picking a runtime by decode rate optimizes for caption-and-elaborate workloads that may not match your use case.

When to Pick MLX vs llama.cpp for Vision

MLX wins when:

• You are running Qwen2.5-VL (3B or 7B) or other Qwen-family vision models
• Short response workloads where encode latency matters
• You are already standardized on MLX for text-only inference and want a single runtime
• You want to integrate via Python and the mlx-vlm package is sufficient

llama.cpp wins when:

• You need LLaVA 1.6 or any architecture that MLX-VLM does not stably support
• You need an OpenAI-compatible HTTP server for vision (llama.cpp's vision endpoint is more mature than mlx-vlm's server)
• You are deploying across mixed hardware including non-Apple platforms and want a single runtime story
• You need GGUF model distribution for offline or air-gapped deployments

Neither runtime is a "best of all" answer right now. The vision model runtime story on Apple Silicon is roughly where text-only inference was 18 months ago: fast-moving, gappy in places, with real performance left on the table by both runtimes.

When This Applies to Your Stack

If you are building product attribute extraction pipelines, internal content moderation tooling, or visual search features on a Mac-based workstation fleet, local vision LLMs are now production viable for the right workload shape. The right shape is short responses, batch-friendly throughput requirements, and inputs that do not require frontier model capability.

If your workload requires dense reasoning over complex images (chart interpretation, multi-document layout understanding, technical diagrams), you will hit a capability ceiling with open weights vision models in the 3B to 13B range and should still route to hosted GPT-4o, Claude Opus 4.8, or Gemini 3.1 Pro vision endpoints for those calls. The hybrid pattern (local for high-volume simple inputs, hosted for hard inputs) is the right architecture for most real product applications.

How to Evaluate This for Your Team

Run your own benchmark with your own image distribution before committing to a runtime. The numbers above are useful as a directional reference but vary substantially by image size, model variant, prompt structure, and macOS version. Build a 100 image evaluation set drawn from your actual workload, measure end-to-end latency and answer quality on each runtime, and pick the configuration that meets your latency budget at acceptable quality.

Contra Collective helps enterprise teams scope and ship local AI inference infrastructure across Apple Silicon and Linux hardware. If you are evaluating local vision LLM deployment patterns for an e-commerce, retail, or content workflow and want a defensible architectural assessment, we are happy to help. Reach out through the services page if you want to talk through your stack.

FAQ

Does MLX support all the same vision LLMs as llama.cpp?

No. As of mid-2026 MLX has strong support for Qwen2.5-VL and Moondream, partial support for several other architectures, and incomplete support for LLaVA 1.6. llama.cpp covers a wider range of vision model architectures but typically lags MLX on Qwen-VL performance.

What is the smallest Apple Silicon Mac that can run a useful vision LLM?

An M2 with 16 GB unified memory can run Moondream 2 at 4-bit quantization with acceptable latency for narrow visual QA tasks. For Qwen2.5-VL 3B at 4-bit you want at least 16 GB to leave headroom, and 24 GB for 7B at 4-bit. Below 16 GB you are limited to Moondream and similarly small models.

Is there an OpenAI-compatible HTTP server for vision on MLX?

mlx-vlm exposes a basic HTTP server but it lags llama.cpp's llama-server for vision endpoints in OpenAI compatibility, multi-image support, and concurrency handling. If you need a drop-in OpenAI vision endpoint replacement today, llama.cpp is the more production-ready option.

Why is encode latency so much higher than decode time per token?

Because encode is a single large ViT forward pass over the entire image, not an incremental autoregressive step. It is a one-shot cost paid per image input, while decode is amortized across the generated token sequence. For short responses, encode dominates total latency.

Can I batch multiple images in a single MLX or llama.cpp request?

Both runtimes support multi-image inputs in a single request but neither has mature batched-request handling across multiple users in the way vLLM does for text-only inference on NVIDIA hardware. For high-concurrency vision serving you currently sacrifice either throughput (sequential processing) or you build batching logic at the application layer.