RecursiveSI/flint 40b-a2.5b
Maybe a better alternative to Tiel Coder for people with 32GB RAM and 8GB VRAM
https://huggingface.co/RecursiveSI/flintOpen linkView original on lemmus.orgSyndicated from the fediverse. Read and engage on the original instance.
View original on lemmus.orghttps://huggingface.co/RecursiveSI/flintOpen linkView original on lemmus.org
4 replies
How do you run a model split between ram and vram? Everything I've read so far says that will tank performance...
Is it something of a, "it works, but prepare to spend a lot of time at the water cooler" type deal, or am I mis informed?
It's a MoE model so it's much more flexible than a dense model, see my guide for llama.cpp https://lemmus.org/post/24235317
Now there's also
--moe-cache-mibthat could beat--n-cpu-moedepending on your exact hardware. I'll need to write an updated guide.And now there's Strata and PolyStrata which get even better performance than that.
This one is a Mixture of Experts model (40B total weight, 2.5B active during token generation). Inference engines like llama.cpp have flags you can pass to offload the lesser used experts to system RAM.
It's slower than having the entire model resident in VRAM, but generally pretty usable.
Looks interesting, don't mind if I yoink