Spyke

Syndicated from the fediverse. Read and engage on the original instance.

View original on piefed.zip
localllama·LocalLLaMAbyTheCornCollector

Llama.cpp MTP Support merged - up to 2.5x speed increase

Qwen3.6-27B-MTP-UD-Q5_K_XL on my 7900XTX goes from 32 t/s to 50-72 t/s depending on the predictability of the task. So, a 1.5x increase on creative tasks up to a 2.2x increase on math.

MTP does not change the quality with the only cost being a few hundred MB extra VRAM usage. You will need to download a gguf model with MTP support to use it.
My parameters:

; Context memory usage  
ctx-size = 65536  
ctk = q8_0  
ctv = q8_0  

; Prompt processing speed  
batch-size = 1024  
ubatch-size = 1024  

; Speculative decoding  
np = 1  
spec-type = draft-mtp  
spec-draft-n-max = 3

Edit: did some more testing using Unsloth's parameters and with spec-draft-n-max = 6 I can get up to 82 tk/s, a 2.56x increase, on the same math prompt. But this comes at the cost of the creative writing task that now falls below 40 tk/s.
It seems like this should be tweaked depending on the prompt similar to the sampling parameters.

Llama.cpp MTP Support merged - up to 2.5x speed increasehttps://github.com/ggml-org/llama.cpp/pull/22673Open linkView original on piefed.zip
45
Comments3

3 replies

Avid Amoeba
lemmy.ca

This does 18tps on 2x R9700:

[Qwen3.6-27B-Q8_0-Code-256K]
m = /models/Qwen3.6-27B/Qwen3.6-27B-Q8_0.gguf
mmproj = /models/Qwen3.6-27B/mmproj-BF16.gguf
chat-template-kwargs = {"preserve_thinking": true}
ctx-size = 262144
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0

This does 39tps on the same hardware:

[Qwen3.6-27B-MTP-Q8_0-Code-256K]
m = /models/Qwen3.6-27B-MTP/Qwen3.6-27B-Q8_0.gguf
mmproj = /models/Qwen3.6-27B-MTP/mmproj-BF16.gguf
spec-type = draft-mtp
spec-draft-n-max = 2
chat-template-kwargs = {"preserve_thinking": true}
ctx-size = 262144
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0

😱

3
robber
lemmy.ml

Using MTP combined with tensor parallelism, I was able to go from running Qwen3.6 27b at ~7t/s to ~30t/s which I think is an insane boost (3x RTX 2000e Ada).

3

You reached the end

Llama.cpp MTP Support merged - up to 2.5x speed increase | Spyke