Ollama: Structured Output Meets Speculative Decoding
The MLX runner now supports speculative decoding and grammar-constrained output at the same time, closing a performance gap that previously forced a choice between the two. Supporting CI and parser fixes round out a release focused on making structured output both faster and more reliable.
Duration: PT2M35S
Episode overview
This episode is a short developer briefing from Ollama.
It explains recent repository work in plain language.
- Show: Ollama
- Published: 2026-09-04T13:01:24Z
- Audio duration: PT2M35S
Transcript excerpt
This excerpt keeps the crawler page concise. Listen to the episode or use the RSS feed for the full update.
This is Ollama, your development briefing for September 4th, 2026.
The headline today: structured output on the MLX runner just got significantly faster, and the fix required rethinking how grammar enforcement and speculative decoding interact.
Jesse Gross's work, spanning commit 4986e92 and PR 18105, tackles a real bottleneck. Previously, a structured-output request couldn't use a model's draft head at all — it fell back to decoding one token at a time, running at roughly half the speed of speculative decoding on a dense 27B MTP model. The fix moves…
This builds on an earlier move, in commit b68365a, to compile structured output as structural tags rather than plain JSON schemas — letting the grammar system express free text, thinking regions, and tool calls in one tree instead of bolting schemas on separately.
Supporting this effort, Daniel Hiltgen's PR 18219 fixes a CI blind spot: MLX test payloads could go stale when the xgrammar wrapper changed or MLX version pins moved, silently skipping tests instead of failing loudly. CI now rebuilds the dylib on drift and builds a Metal variant when no matching release exists —…
Separately, PR 18216 from pentaoa fixes a correctness issue in…
Nearby episodes from Ollama
- Hardening the MLX Engine and Fixing What Feeds the Scheduler
- MLX Grows Up, Model Intent Gets Respected
- Runner Reliability and a Documentation Cleanup Wave
- Proxy Fix and a Documentation Cleanup Sweep
- Weekly Recap - Claude Desktop Integration & MLX Engine Maturity
- Correctness Fixes Under the Hood
- Agent Expansion, Desktop Polish, and a Documentation Sweep
- MLX Engine Grows Up