The recent integration of Multi-Token Prediction (MTP) into llama.cpp has sparked excitement among developers, especially those using the Qwen3.6 27B model. Community benchmarks show that local inference speeds have significantly improved, with gains reported at up to 2.4 times on AMD Strix Halo systems. This enhancement is especially important for open-weight models, which are now more viable for tasks requiring low latency and increased privacy.
The merge took place on May 16 through PR #22673, bringing MTP speculative decoding into the main branch of llama.cpp. Early tests indicate that Qwen3.6 27B can generate tokens at a notably faster rate, especially when paired with MTP-capable GGUF weights. This advancement could transform user experiences in chat, coding, and agent workflows, where lower latency promotes more interaction and experimentation.
A key feature of this update is its accessibility. MTP enables supported models to draft multiple tokens ahead and explore paths through speculative decoding without the need for more powerful GPUs. This approach maximizes the potential of existing consumer-class GPUs, making local AI applications more feasible for a wider range of developers.
Performance Benchmarks and Community Feedback
Community benchmarks highlight the performance improvements brought by MTP. For example, one user reported an increase from 7.4 to 18.1 tokens per second on an AMD Strix Halo, achieving a 2.44x speed boost. A dual RTX 3090 setup experienced improvements from 25.7 to 55.9 tokens per second, while a single RTX 3090 improved from 38.7 to 59.5 tokens per second. These results suggest that the speed gains are substantial enough for use beyond controlled lab environments.
Additionally, a tutorial published shortly before the MTP merge indicated that enabling this new feature on an RTX 3090 improved throughput from 38 tokens per second to 65 tokens per second, representing a 1.71x speedup. While not every system may see a doubling in performance, these enhancements are significant enough to attract developers' attention.
Implications for Local AI Development
The implications of these developments go beyond performance metrics. For local AI users, the improved speed means previously useful but sluggish models can now be integrated into daily workflows more efficiently. Reduced latency increases user engagement, encouraging more follow-up questions and additional local experiments. This shift reduces reliance on hosted inference, especially in situations where privacy and cost are critical.
However, while MTP shows promising results on dense models, its effectiveness is reduced on mixture-of-experts models. Tests revealed that Qwen3.6 35B-A3B demonstrated smaller gains, as only a portion of the model is active for each token generated. This distinction underscores that while MTP offers significant advantages in specific contexts, it is not a one-size-fits-all solution for all model architectures.
As local AI continues to develop, features like Multi-Token Prediction are likely to play an important role in making AI development more accessible and efficient. By enabling faster token generation on existing hardware, llama.cpp is establishing a new standard in local inference, making AI tools more practical for everyday use. The growing community support and feedback will be crucial in refining these advancements to meet the diverse needs of developers and end-users alike.
Quick answers
What is Multi-Token Prediction?
Multi-Token Prediction is a feature that allows models to draft multiple tokens ahead and use speculative decoding, enhancing inference speed.
How does MTP affect the performance of Qwen3.6 27B?
MTP has resulted in performance improvements of up to 2.4 times for local inference, making it faster and more efficient on consumer-grade GPUs.
Can MTP be used with all AI models?
While MTP shows strong improvements with dense models, its effectiveness is reduced for mixture-of-experts models.
