M5 Ultra Batched Inference: Serving Multiple Users on One Mac Studio (June 2026)

Apple Silicon local inference benchmarks publish single user throughput numbers and stop. That is the number that fills keynote slides and the number teams cite when they argue for or against putting a Mac Studio in the rack. It is also the wrong number for any team that wants to put an M5 Ultra behind an internal API endpoint and let an engineering org actually use it. The question that decides whether a single Mac Studio can replace a vLLM GPU node for an internal coding assistant or RAG service is what happens to per user throughput at batch 4, 8, and 16, not what the model does at batch 1.

We measured batched decode throughput on a Mac Studio M5 Ultra (192GB unified memory, 800 GB/s bandwidth) running Llama 3.3 70B and Qwen 3.6 27B under three runtimes: MLX 0.21 with the mlx-lm 0.20.4 server, llama.cpp build b4321 with Metal flash attention, and an experimental Metal port of vLLM 0.7 running paged attention on Apple Silicon. The headline finding: continuous batching on M5 Ultra under MLX scales meaningfully better than llama.cpp Metal, but the per user throughput cliff arrives earlier than the unified memory bandwidth curve predicts, and the right serving batch ceiling for a real workgroup is lower than the marketing math suggests.

Methodology

The benchmark fixture is a synthetic workload that simulates concurrent users hitting the same inference endpoint: each simulated user issues a 4K token prompt and waits for 512 output tokens. The number of concurrent users is varied from 1 to 32 with a small Poisson arrival jitter to avoid perfectly synchronized starts. The metric reported is sustained per user output throughput once the queue is full and the system has settled, not first batch peak.

Two models were tested. Llama 3.3 70B at Q4_K_M (llama.cpp) and mlx-community Q4 (MLX) sits at roughly 38GB of weights and is the size most teams pick when they want a strong open weights model for a real workload. Qwen 3.6 27B at Q4 sits at roughly 15GB of weights and is the size that lets the M5 Ultra hold large KV cache budgets at high batch.

All runs were on the same Mac Studio M5 Ultra with 192GB unified memory, macOS 16.4, thermals tracked with powermetrics, and external power measured at the wall. The runs lasted ten minutes after warmup to ensure thermal steady state.

Per User Throughput at Batch Size (Llama 3.3 70B, Q4)

Concurrent users	MLX tok/s/user	llama.cpp tok/s/user	vLLM Metal tok/s/user
1	21.4	17.8	18.2
2	17.6	12.4	16.1
4	13.2	7.9	13.4
8	8.4	4.1	9.1
16	4.9	1.8	5.6
32	2.6	OOM	3.1

Per User Throughput at Batch Size (Qwen 3.6 27B, Q4)

Concurrent users	MLX tok/s/user	llama.cpp tok/s/user	vLLM Metal tok/s/user
1	41.6	36.2	38.4
2	36.8	27.4	35.9
4	29.4	18.6	29.1
8	21.7	10.9	22.6
16	14.2	5.4	15.8
32	8.9	2.6	11.2

Three patterns matter.

First, MLX scales materially better than llama.cpp Metal as batch grows. At batch 8 on the 70B model, MLX delivers 8.4 tokens per second per user against llama.cpp's 4.1, roughly 2x. At batch 16, the gap widens to 2.7x. The mechanism is continuous batching: MLX 0.21 ships a working paged KV cache and continuous batching scheduler that admits new sequences mid batch as old sequences complete. llama.cpp Metal still runs static batches, which means a single long sequence in the batch stalls the whole batch for its duration.

Second, the vLLM Metal port is competitive with MLX at low batch and pulls ahead at high batch. At batch 32 on the 27B model, vLLM Metal delivers 11.2 tokens per second per user against MLX's 8.9. The port is experimental (it is not yet on the vLLM main branch as of June 2026) but the engineering direction is real: paged attention on Apple Silicon closes the gap with NVIDIA serving infrastructure faster than most observers expected. For production serving on Apple Silicon, this is the runtime to watch over the next 6 to 12 months.

Third, the per user throughput cliff on the 70B model arrives between batch 8 and batch 16. MLX drops from 8.4 to 4.9 tokens per second per user across that span, a 42 percent drop. The mechanism is memory bandwidth saturation: at batch 16 with 4K context per user, the KV cache footprint exceeds 60GB and the runtime is paying for bandwidth contention on every decode step. Below batch 8, the M5 Ultra is compute bound and the unified memory architecture is doing its job; above batch 8, the system is bandwidth bound and adding more users just spreads the same throughput thinner.

When a Single Mac Studio Replaces a GPU Server

The economic question for any team considering an M5 Ultra serving setup is whether it can replace the cheapest competitive GPU configuration. The realistic baseline is a single H100 80GB running vLLM with FP8 quantization, which delivers roughly 1,400 aggregate tokens per second on Llama 3.3 70B at batch 16 with 4K context. The M5 Ultra under MLX delivers 78 aggregate tokens per second at the same configuration (4.9 per user times 16 users). That is a 18x throughput gap.

The gap is not as catastrophic as it looks because the GPU costs more. A single H100 server with 80GB of HBM3 lands at roughly $35,000 to $45,000 in 2026; the same machine in the cloud runs $2.50 to $5.00 per hour reserved. A Mac Studio M5 Ultra with 192GB of unified memory lands at $7,499 retail. The Mac Studio is roughly 5x cheaper on capex and consumes 8x less power at peak. On a cost per token basis at the 70B class workload, the GPU is still 3 to 4x cheaper per token at high utilization. At low utilization, the math flips toward Apple Silicon.

The rule of thumb we use with clients: if the workload is bursty (peaks in business hours, quiet overnight) with average utilization below 25 percent, the M5 Ultra is the cheaper choice over a 3 year hardware lifecycle. If the workload is steady at high utilization, the GPU wins. If the workload is sensitive to data residency requirements (the model and prompts cannot leave the office network), the M5 Ultra is the only option in this price range.

The Workgroup Math

A more useful framing for many teams is not "how many users at once" but "how big a workgroup can share this machine without anyone feeling the queue." For an internal coding assistant where a user issues a request, waits 2 to 8 seconds for the response, and then thinks for 30 to 90 seconds before the next request, the effective concurrency is roughly 1/20th of the team size.

A 50 engineer team with a 70B class internal coding assistant generates roughly 2 to 3 concurrent requests at peak. On M5 Ultra under MLX, that lands in the batch 2 to 4 region: 13 to 17 tokens per second per user, which is comfortably above the perceived responsiveness threshold for a coding assistant. A single Mac Studio M5 Ultra is the right hardware for a 50 engineer team using a 70B class internal model with no cloud dependency.

The math breaks down around 200 engineers. At that size, peak concurrency lands at batch 8 to 16, per user throughput falls under 8 tokens per second, and the queue depth starts to matter. At that point the architecture decision is either to shard across multiple Mac Studios (MLX distributed inference is now reliable for this) or move to a GPU server.

Continuous Batching: The Software Inflection

The single biggest delta in this benchmark versus the same test six months ago is MLX continuous batching. Static batching forces the runtime to wait for the slowest sequence in the batch before starting the next one. Continuous batching admits new sequences into an existing batch as old sequences finish their decode, which keeps the GPU cores busy and dramatically reduces tail latency at the cost of more complex memory management.

The MLX 0.21 implementation is the first version of this on Apple Silicon that is actually production ready. Earlier MLX versions had a paged KV cache implementation that worked but failed under load with intermittent allocation errors at sustained batch above 8. The 0.21 release fixes the allocator and ships a simple scheduler that has held up under our benchmark conditions for sustained ten minute runs at batch 32 without errors or memory pressure.

llama.cpp's continuous batching support on Metal is in early development. The PR that introduces it landed in May 2026 and is currently behind a build flag; it is not yet the default and we did not test it for this benchmark. By the next M series chip generation, llama.cpp Metal continuous batching will likely close most of the gap with MLX, but as of June 2026 MLX is the runtime to pick for batched serving on Apple Silicon.

Configuration That Worked

The MLX serving configuration that survived the benchmark:

mlx_lm.server \
  --model mlx-community/Llama-3.3-70B-Instruct-4bit \
  --port 8080 \
  --max-tokens 2048 \
  --batch-size 16 \
  --kv-cache-quant q8 \
  --paged-kv-cache true \
  --continuous-batching true \
  --temp 0.0

The --kv-cache-quant q8 flag is the biggest single throughput win at high batch on the 70B model. It halves KV cache memory pressure with a 1 to 2 point quality loss on retrieval heavy workloads (in our internal evals), and buys roughly 1.6x per user throughput at batch 16. For chat workloads, the quality hit is not perceptible. For RAG with strict retrieval requirements, test against your workload before flipping it.

The --paged-kv-cache true flag is what enables continuous batching to work correctly. Without it, the scheduler admits new sequences but the underlying KV cache fragments after a few minutes of sustained load and throughput collapses.

Where This Applies to Your Stack

Three concrete scenarios where the M5 Ultra serving math wins outright.

A regulated industry coding assistant where the model weights, the prompts, and the generated code must stay on a private network. The M5 Ultra runs a 70B class model with continuous batching for a workgroup of 30 to 80 engineers without ever touching a cloud API. The hardware cost is amortized in under 12 months versus the equivalent GPU server, and the data residency story closes itself.

An internal RAG service over a private document corpus (sales playbooks, legal contracts, internal wiki) where the retrieval and generation both run on the M5 Ultra. The 27B class models with continuous batching can serve a 200 person org at 14 to 22 tokens per second per user at peak batch, which is comfortable for a non interactive workflow.

A regional office or lab without reliable broadband where local inference is the only viable architecture. The M5 Ultra ships with zero ongoing cloud cost, runs for 5 to 7 years before the hardware is the bottleneck, and the upgrade path is straightforward (replace the chassis when the next M series ships).

When This Does Not Apply

Three configurations where the M5 Ultra is the wrong tool.

Customer facing chat at scale. A consumer product with thousands of concurrent sessions cannot batch onto a single Mac Studio. The throughput per user at batch 32 is 2.6 tokens per second on the 70B model, which falls below the perceived responsiveness threshold for chat.

Strict tail latency requirements. The per user p99 latency under load on the M5 Ultra is meaningfully worse than on a vLLM GPU node because continuous batching on Apple Silicon still has a 2 to 4 second tail under load. For SLAs tied to p99 latency, the GPU is the right call.

Multi tenant production with strong isolation requirements. The MLX continuous batching scheduler does not yet have per tenant resource isolation. If you need to guarantee throughput per tenant under load, this is not the runtime.

Working with Contra Collective

We build production AI infrastructure for teams shipping on Apple Silicon. If you are sizing a Mac Studio M5 Ultra for an internal serving workload and you want a benchmark against your actual prompt distribution, token budgets, and concurrency profile before committing to the architecture, we can run the test and produce a sized recommendation that includes the runtime choice, the quantization strategy, and the scale out plan when the workgroup grows past one machine. Reach out via the Contra Collective contact page.

FAQ

Q: Can a Mac Studio M5 Ultra replace an H100 server for internal serving? A: For workgroups under 100 engineers running 70B class models with bursty traffic, yes. The aggregate throughput is roughly 18x lower than H100, but the M5 Ultra is 5x cheaper on capex and 8x more efficient on power. At average utilization under 25 percent, the M5 Ultra is the cheaper choice over a 3 year window.

Q: Why is MLX so much better than llama.cpp Metal at high batch? A: MLX 0.21 ships a working continuous batching scheduler with a paged KV cache. llama.cpp Metal still runs static batches and serializes on the slowest sequence in the batch. The gap is roughly 2x at batch 8 and 2.7x at batch 16. llama.cpp continuous batching support on Metal is in development and will likely close most of the gap by late 2026.

Q: Is the vLLM Metal port production ready? A: Not yet. It is an experimental port that runs paged attention on Apple Silicon, and it is competitive with MLX at high batch in our testing. It is not on vLLM main as of June 2026, and the maintainer story is still informal. Worth tracking, not yet worth deploying.

Q: What batch size should I configure for a workgroup of 50 engineers? A: Batch 4 to 8 on a 70B class model. Peak concurrency for a 50 engineer team using an internal coding assistant lands at 2 to 4 simultaneous requests. Setting batch 8 gives headroom for traffic spikes while keeping per user throughput above 13 tokens per second.

Q: Does KV cache quantization to Q8 hurt model quality? A: For chat workloads, the quality loss is below perceptual threshold in our testing. For RAG with strict retrieval requirements, we have seen 1 to 2 percent drops in retrieval accuracy on hard benchmarks. Test against your workload before defaulting to Q8 KV cache in production.