what is your selfhosted local LLM stack?
What tools and projects do you use for your local LLM chats and agents? I'm trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.
Syndicated from the fediverse. Read and engage on the original instance.
View original on lemmy.blahaj.zoneView original on lemmy.blahaj.zone
8 replies
llama.cpp built for hip/rocm (since I have AMD GPUs) and a bunch of custom code. One of the first things I did with LLMs was make my own chat client. Most of the useful stuff I get out of LLMs right now involves small focused tasks that I run in a batch with custom scripts (e.g. bulk OCR / image analysis tasks and stuff like that).
Without knowing your hardware, it's impossible to give you advice.
I'm on an 8gb rx7600 with a 5700x CPU with 64gb 3600mts ram.
I use llama.cpp with the rocm driver, then have a custom sandbox for coding and heavily restricted filesystem access, all setup in pi harness, with access to a searxng instance for web search. Using orinth1.5 35b a3b on the iq4_xs unsloth quant.
This gives me the full ability to do anything, from creating reports, coding projects and fairly complex tasks.
I also have stable diffusion setup for image generation with flux 2 klein 4b, which is using qwen4 4b.
Hermes agent (actually two: one running Qwen3.8-27b locally via llama.cop on an RTX 3090 running Ubuntu server with access to my personal stuff as an assistant but no internet access to avoid personal info or secrets egress and another running Deepseek 4.1 Flash 1017/GPT-6.1 Sol - for tougher tasks - served via OpenRouter with full web access but no access to personal info and secrets). They both run in docker containers with limited access to host. I also use squid or caddy reverse proxies to protect against unauthorized web or network access. I am the bridge between them when needed.
I've been playing with small models that can fit on the pitiful 8gb of VRAM on my Radeon RX 7600. So far qwen3.5-9b-artemis has done pretty well at solving merge conflicts on rebases which involves some tool calling. I might try some of the other tooling tuned versions if anyone has any recommendations.
I didn't have much joy with any of the Gemma4 models but that might have been due to API differences.
I'm on an rx7600 too. I have done loads of testing and benchmarking and the best model is orinth1.5 35b a3b. I get around 40tok/s generation from it. I run it with llama.cpp with the rocm driver, and use the iq4_xs unsloth quant. It's based on qwen3.6 35b a3b, but trained on complex thinking and took calling. It's miles ahead of anything else I've tried.
How big is your system prompt? My 6.5k (which is mostly 5.5k of tool defs) leads to an error message.
Not at my pc now, but it's fairly big. I use pi harness, with a searxng instance, plus a custom sandbox and a read only mount point and a full write mount point plus office tools and coding tools. I use a context window of 130k tokens with auto compaction.
Ollama running Llama and Gemma, alongside Whisper for speech recognition, running off a 4070 12GB.
Gemma does not work quite right and probably can't stay, just not quite enough memory even when it's the only model loaded.