Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
Crossposted from https://lemmy.ml/post/47429470
https://github.com/chiennv2000/orthrusOpen linkView original on mander.xyzSyndicated from the fediverse. Read and engage on the original instance.
View original on mander.xyzhttps://github.com/chiennv2000/orthrusOpen linkView original on mander.xyz
2 replies
My oversimplified and possibly wrong understanding: this is like speculative decoding, but instead of a separate draft model (which does its own prompt processing), they use some diffusion thing strapped on top of the main model. The diffusion reuses the high-quality prompt processing result of the main model.
The 7.8x faster claim sounds almost too good to be true. But even if we get like 3x then this is still a huge revolution in localLLMing.
They said they're working on Orthus for Qwen 3.5. It'll be amazing!