Perplexity Lily vs MLX-LM: Qwen3.6 Local Inference on M5 Max, Tested (2026)

Perplexity just open sourced Lily, the local inference engine it built for hybrid compute in Perplexity Computer, and it is a clean demonstration of what specialization buys you on Apple Silicon. Lily is not a general runtime. It runs one model family, Qwen3.6-35B-A3B, on Apple hardware, and it is built so on device compute does not bottleneck the tasks Perplexity Computer hands off locally. On an M5 Max MacBook Pro, Perplexity reports average throughput gains over MLX-LM of 1.23x on prefill and 1.35x on decode, with output quality effectively unchanged. Same chip, same model, same quantization, and a third more decode throughput from nothing but a better engine. That is worth understanding, because the reason it works generalizes even if the engine does not.

Attribute	Perplexity Lily	MLX-LM
Scope	one model, Qwen3.6-35B-A3B	general, most MLX models
Implementation	Rust host plus hand written Metal kernels	Python plus MLX framework
Prefill throughput (4K/4K, M5 Max)	5,749.9 tokens per second	4,737.5 tokens per second
Decode throughput (4K/4K, M5 Max)	186.6 tokens per second	140.9 tokens per second
Prefill speedup	1.23x	baseline
Decode speedup	1.35x	baseline
API	OpenAI compatible	varies by wrapper

Those throughput numbers are Perplexity's reported figures at a 4K prompt with a 4K decode context, and they are the real basis for the speedup claims rather than round numbers.

What Lily actually is

Lily is a single process runtime with a clear division of labor. A Rust layer loads the checkpoint and drives the generation loop, hand written Metal kernels execute the model on the GPU, and an OpenAI compatible chat completions API streams tokens back to the caller. There is no Python in the hot path, which removes a class of overhead, and the kernels are written for this model rather than generated by a general framework. The 4 bit checkpoint is 19.4 GB, so the realistic floor is an Apple Silicon Mac with 32 GB or more of unified memory. Because the endpoint is OpenAI compatible, dropping it into an existing agent stack is trivial:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

resp = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[{"role": "user", "content": "Summarize this repo's build steps."}],
    max_tokens=512,
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="")

The Qwen3.6-35B-A3B target matters. The A3B suffix means roughly 3B active parameters out of 35B total, a sparse mixture of experts. That is a model that is memory heavy to hold but light to compute per token, which is exactly the profile where a tightly written engine can extract the most, because the bottleneck sits in memory movement and kernel efficiency rather than raw arithmetic. If you want the background on how memory bandwidth governs local throughput on these chips, we covered it in Apple Silicon memory bandwidth and tokens per second on M5.

Why splitting prefill and decode buys 1.35x

The core design choice is that Lily treats prefill and decode as fundamentally different workloads and optimizes them separately. This is the part worth internalizing, because it is true regardless of which engine you run.

Prefill processes many prompt tokens at once. Because all those tokens are available up front, the engine can reuse loaded weights across them, so prefill is compute bound and parallel friendly. Decode is the opposite. It generates one token at a time, each depending on the last, so there is almost no reuse and the engine reloads weights from memory for every single token. Decode is memory bandwidth bound. On a sparse model like Qwen3.6-35B-A3B, decode is where most local setups bleed throughput, because you are paying memory traffic for a large model to emit one token at a time.

A general runtime like MLX-LM handles both phases with the same broad machinery, which is the right call when you must support hundreds of model shapes. Lily gives up that generality and writes each phase for this one model on this one hardware family, mapping the Qwen operations onto the M5's compute and memory layout by hand. The larger decode speedup, 1.35x against 1.23x on prefill, is the tell: the biggest wins come from the memory bound phase, which is precisely where careful kernel work and cache behavior pay off. We compared how the general engines trade off across this axis in llama.cpp versus MLX, Ollama, and vLLM on Apple Silicon.

What you give up

Specialization is a trade, not a free lunch. Lily runs one model family. The moment you want a different model, a different size, or a different quantization scheme, you are back to a general runtime. MLX-LM runs essentially the whole MLX ecosystem, swaps models with a config change, and keeps pace with new architectures as they land. Lily keeps pace with exactly one. For quantization flexibility specifically, the format you choose still matters as much as the engine, which we broke down in GGUF versus MLX quantization formats on Apple Silicon.

So the honest framing is not that Lily beats MLX-LM. It is that a hand tuned engine for one model beats a general engine running that same model by 20 to 35 percent, at the cost of running only that model. If your product pins a single model and ships it to many Macs, that trade is excellent, and it is exactly why Perplexity built Lily for Perplexity Computer. If you need to run whatever model this month's benchmark favors, the general runtime's flexibility is worth more than a fixed one third on decode.

When a narrow engine is the right call

The pattern to take from Lily is that at the edge, where you control both the model and the hardware, specialization is an underused advantage. A team shipping a fixed local model to a known Apple Silicon target can often find a similar 20 to 35 percent by writing to the metal rather than accepting a general framework's defaults, and that headroom can be the difference between a responsive on device experience and one that feels sluggish. The catch is maintenance: a hand written engine is code you now own and must carry forward as models and chips change, so it only pays when the model is stable and the volume of devices justifies the engineering.

You do not have to write your own to benefit, either. Lily is open source, its endpoint is OpenAI compatible, and if your product happens to run Qwen3.6-35B-A3B on Apple Silicon, you can adopt the speedup directly. That is the more common path: use the specialized engine when it matches your model, keep a general runtime for everything else.

FAQ

What is Perplexity Lily? Lily is an open source local inference engine from Perplexity, written in Rust with hand written Metal kernels, specialized for Qwen3.6-35B-A3B on Apple Silicon. It exposes an OpenAI compatible API and was built for hybrid compute in Perplexity Computer.

How much faster is Lily than MLX-LM? On an M5 Max at a 4K prompt and 4K decode context, Perplexity reports 1.23x higher prefill throughput and 1.35x higher decode throughput, with output quality effectively unchanged. In raw numbers that is 5,749.9 against 4,737.5 prefill tokens per second and 186.6 against 140.9 on decode.

What hardware do I need to run it? The 4 bit checkpoint is 19.4 GB, so a realistic floor is an Apple Silicon Mac with 32 GB or more of unified memory. More headroom helps once you add context and other applications.

Why is the decode speedup larger than the prefill speedup? Decode is memory bandwidth bound because it generates one token at a time with little weight reuse, so careful kernel and memory work pays off most there. Prefill is more compute bound and already parallel, leaving less to gain.

Should I replace MLX-LM with Lily? Only if your product runs Qwen3.6-35B-A3B on Apple Silicon and you want the speedup. Lily runs one model family, so for anything else, or if you swap models often, keep a general runtime like MLX-LM.

When this applies to your stack

If you are shipping local inference on Apple Silicon, Lily is a concrete reminder that the engine is a tuning knob, not a fixed cost. When you pin a model and a hardware target, a specialized runtime can hand you 20 to 35 percent on the same silicon, and on the memory bound decode phase that gap is often the difference between an experience that feels instant and one that does not. When you need to run many models, the general runtime's flexibility is the better buy, and you accept its defaults as the price of that reach.

Deciding which side of that trade your product sits on, and building the local serving path to match, is the kind of infrastructure work we do for clients running AI on the edge. If you are standing up local inference on Apple Silicon and want to know whether a specialized engine is worth the maintenance or whether a general runtime is the disciplined choice, Contra Collective can benchmark your model on your target hardware and build the deployment around the answer.