I'll have to do an updated guide once Llama.cpp v0.7.0 comes out because there's a new option now for MoE expert caching. It doesn't work that well yet but I think they'll continue to improve it. https://lemmus.org/post/25955310
Benchmarks like SWE-Bench Pro are pretty close to Qwen 3.8 27B. I suppose this model is about what a Qwen 3.8 35b-a3b model would be. Alibaba appears to have moved onto Qwen 4 and haven’t announced any 35b-a3b models, so I’m glad this exists.
I don't really understand your comment? Qwen3.8 27b is qwen4 architecture as a pre-release. apparently i was wrong here, it's still on the 3.5 architecture.
A 27b dense model is really difficult to compare to a 35b a3b Moe model. I tried to run qwen3.8 27b on my 8gb card and ok it runs technically, but in no way is it viable or does it come close to ornith 1.5 35b a3b, mostly because to run qwen3.8 27b, I have to choke it to almost useless levels by running an aggressive quant like a 1, 2, or 3 bit quant, and even then it doesn't run as fast.
FYI: Qwen3.8 27B is still a Qwen3.5 architecture model (Qwen3_5ForConditionalGeneration in config.json); Qwen3.8-Flash-Next -- a different model, despite the similar name -- is the Qwen4 preview (Qwen4ExpForConditionalGeneration).
I believe what they're saying is that since we didn't get a MoE around the 35B-A3B size for the Qwen3.8 generation of releases that this model might be able to fill that niche -- i.e. a better model than Qwen3.6 35B-A3B while still being less compute intensive to run than Qwen3.8 27B.
Yeah, exactly, and sorry to everyone if I was unclear. I’m not holding out hope for a Qwen 4 35b-a3b either, so it’s good to see a model that comes close to 3.8 27b in benchmarks and I hope they stay competitive with Qwen 4 27b as well.
Yes they're based on Qwen. Ornith 1.5 is a fine-tune of Qwen.
Tiel Coder is based on Ornith 1.5, it's technically just a quant and an improved template, but it seems like an improvement for people who are going to use Q4 anyways.
I got a 35B model that runs at 20 tok/s with a 130K context window on my 8GB card. Can be bumped up to 200K if I want, with marginally slower output speeds.
16 replies
But is it actually better than Ornith or Tiel 35b-a3b?
I can do 80k+ just fine with 35b-a3b. I don't think 50k would be enough for agentic coding.
And you too have 8GB of VRAM? Could you please tell me where can I find such configuration to try?
Yeah RTX 2080 8GB and 32GB of system RAM, I wrote a guide
https://lemmus.org/post/24235317
I'll have to do an updated guide once Llama.cpp v0.7.0 comes out because there's a new option now for MoE expert caching. It doesn't work that well yet but I think they'll continue to improve it. https://lemmus.org/post/25955310
Thanks a lot 🙇🙇🙇🙇
I run ornith 1.5 35b a3b on my 8gb Rx 7600, get 30 t/s, and have context at 130k. It slows down to around 15 t/s near the end on the context window.
Nothing I have tested comes close to this model.
I use it in pi with search, coding sandbox and limited filesystem access.
I cap mine to 80k (Q8_0) just cause it seems to be enough for agentic coding (Zoo Code), and past 80k the speed and quality start to drop
I will cap mine for shorter chats to 32k, but for full blown complex tasks, I'll turn it up to 130k. I run the q4 k m imatrix version
Benchmarks like SWE-Bench Pro are pretty close to Qwen 3.8 27B. I suppose this model is about what a Qwen 3.8 35b-a3b model would be. Alibaba appears to have moved onto Qwen 4 and haven’t announced any 35b-a3b models, so I’m glad this exists.
I don't really understand your comment?
Qwen3.8 27b is qwen4 architecture as a pre-release.apparently i was wrong here, it's still on the 3.5 architecture.A 27b dense model is really difficult to compare to a 35b a3b Moe model. I tried to run qwen3.8 27b on my 8gb card and ok it runs technically, but in no way is it viable or does it come close to ornith 1.5 35b a3b, mostly because to run qwen3.8 27b, I have to choke it to almost useless levels by running an aggressive quant like a 1, 2, or 3 bit quant, and even then it doesn't run as fast.
FYI: Qwen3.8 27B is still a Qwen3.5 architecture model (
Qwen3_5ForConditionalGenerationin config.json); Qwen3.8-Flash-Next -- a different model, despite the similar name -- is the Qwen4 preview (Qwen4ExpForConditionalGeneration).I believe what they're saying is that since we didn't get a MoE around the 35B-A3B size for the Qwen3.8 generation of releases that this model might be able to fill that niche -- i.e. a better model than Qwen3.6 35B-A3B while still being less compute intensive to run than Qwen3.8 27B.
AHH cool, I didn't thought both 3.8 models were on 4 architecture, I'll edit my previous comment 👍
Yeah, exactly, and sorry to everyone if I was unclear. I’m not holding out hope for a Qwen 4 35b-a3b either, so it’s good to see a model that comes close to 3.8 27b in benchmarks and I hope they stay competitive with Qwen 4 27b as well.
Ormit? Tiel? Can you elaborate? Are they based on qwen 35b A3B?
Yes they're based on Qwen. Ornith 1.5 is a fine-tune of Qwen.
Tiel Coder is based on Ornith 1.5, it's technically just a quant and an improved template, but it seems like an improvement for people who are going to use Q4 anyways.
I got a 35B model that runs at 20 tok/s with a 130K context window on my 8GB card. Can be bumped up to 200K if I want, with marginally slower output speeds.
I'll have to try this out. I couldn't get Microsoft's ternary bitnets running on llama.cpp or ollama.