Ollama: Speculative Decoding and the Cloud Model Nudge
Jesse Gross landed a multi-commit push to make the MLX engine faster through MTP speculative decoding and kernel fusion, while a separate cluster of PRs from Devon Rifkin closes a UX gap where cloud-only models silently fail to run.
Duration: PT2M25S
Episode overview
This episode is a short developer briefing from Ollama.
It explains recent repository work in plain language.
- Show: Ollama
- Published: 2026-08-01T13:00:50Z
- Audio duration: PT2M25S
Transcript excerpt
This excerpt keeps the crawler page concise. Listen to the episode or use the RSS feed for the full update.
Good morning. It's August 1st, and this is Ollama.
The headline today is speed. Jesse Gross shipped MTP speculative decoding for Qwen 3.5 in the MLX engine, PR 17454, and backed it with a run of supporting commits. The numbers are notable: on an M5 Max, the 35B MoE model jumps about 24% faster on coding prompts, and the dense 27B model gets roughly 70% faster. This…
Second theme: cloud model discoverability. When a model only publishes a cloud tag — kimi-k3 is the example — running or pulling it by default used to fail with a bare file-not-found error. Devon Rifkin's PR 17483 adds a prompt suggesting the cloud variant when that happens, and follow-on PRs extend the same…
Elsewhere: two parser panics got fixed — a cogito tool-call parser that dropped completed calls before the closing delimiter, and a function-call parser that panicked on a lone escape delimiter. Both are the kind of edge case that silently breaks agent tool use. Also landed: a race-condition fix in the updater's…
What to watch: the MLX context-length hang report is still open and worth tracking against the new speculative decoding path. That's the briefing.
Nearby episodes from Ollama
- API Compatibility and Model Correctness Push
- Speculative Decoding Gains and a Lint Lockdown
- Cleaning Up Concurrency and Cutting Experimental Code
- Trust Your Cache, Fix Your Logs
- Parsing Bugs and Process Hardening
- Weekly Recap - New Model Support & Concurrency Hardening
- Closing the Gaps Between Local Config and Cloud Compatibility
- MLX Performance Push and a Scheduler Race Fix