Is anyone self-hosting their own AI?
Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you're open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don't absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I'm not comfortable using them to help me with web searches and the such, because there's no way that all this information won't be going into a profile. This makes me want to have my own "helper" service.
193 replies
Llama.cpp running qwen 3.8 on a mini PC with 32GB ddr5 ram and 8GB vram. Not the most speedy but it works. I only really access it while at home but I'm also set up with tailscale.
So you're using CPU offloading? Or the model's Bonsai2 variation?
I wouldn't mind running a model at home, mostly to help document my setup. But I just don't trust it. So no.
It depends on your hardware. But right now the "cheapest" hardware you can buy for a usable setup, is the strix halo (128GB), and run gufo with qwen 3.8 flash next (Q4XL). It is sonnet 5 level. Very good speed (both prefill 1000+ and decode 50+), with up to 256k context.
I have it set up with omp + searxng. Sometimes with hermes. Really good! It changes the game for local setup.
googles
Jesus christ
Just get the cheapest OEM, Bosgame M5. I know "cheapest" is in quotes. But again, it is the cheapest 128GB unified vram device you can get right now. I got mine at pre-orders, so yes, more than a year and a half ago, prices were a steal. Also, great desktop (mini) PC overall and very efficient, even if you don't want to do AI.
I have like 150GB of DDR3 spread over multiple 2/4/8/16GB sticks, I wonder if I can get that working for something like this 🤔
Why wouldn't you trust a local model? I mean sure they are less intelligent but if you dont expose it publicly you should be good to go
I don't trust it to be accurate; I'd have to reference the sources it pulled from. Extra work for little payoff.
Got it, makes sense. You'd probably have to do some benchmarking first. Depending on how large the source catalog is.
A quick glance shows that I'm in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I'm running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I'm happy to have paid a "mere" $2400 compared to what similar systems cost now, and now I'm toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).
Hermes Agent does a lot of random stuff for me right now. It manages my calendar and to-do list in nextcloud via MCP server. It created and maintains an llm-wiki based on my Obsidian Vault. Then it creates a report every morning about both and the weather.
The server has tailscale installed, so yes I can access it anywhere.
Try gufo with qwen 3.8 flash next. Much much better than 3.6-35B.
How well does it handle concurrency? I could see this as a vast improvement on llamacpp, but I was already speccing out vLLM.
I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it's been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.
I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode With MTP, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine higher decode speeds are possible with some tinkering, but I haven’t gotten around to it.
OpenCode recently got hacked and advises changing API keys fyi.
I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit).
What vision model do you use?
How much electricity does it burn? And how much does it cost to let it run overnight? I feel like this is the same problem that the companies have, its way too expensive to use it on a lots of things and it is way cheaper to use it on someone else's server..
It uses 261 W, probably more expensive than getting a 2nd Mistral subscription with the cost of power where I live (Belgium), but it will be free heat once I have solar panels and a battery.
Thank you, that is not as bad as I expected. 👍
If you are playing around at home I suggest seeking a MoE model. You get less penalty from having less vram and offloading to normal ram. I run Qwen 3.6 quant 6 on 250k context on a 16gb vram 64gb ram ddr4 PC. I get about 31tps which is enough for my use. After a little tweaking I have put results of it's output into Claude Pro for vetting and it hasn't found any flaws in the simple scripts and things I've asked it to make. I can only run Qwen 3.8 which is a dense model and not MoE. At low quant with not as much context. It spits out accurate short context answers if that's all you need. So I prefer 3.6. Tiel is faster but Claude picked up more errors in what I was doing. Orinith kept getting into thinking loops for me so I stopped using it. Anyway I'm just a little guy.
I just bought a 32GB M6 Mac Mini that I'll also use for local ai. Will probably use Qwen3.8-27B. Accessing it from outside my lan would happen via Tailscale.
Even ignoring all of the functional issues with LLMs/"AI" related to their design to never admit to not knowing and preferring to make stuff up instead (hallucinate), I have an ethical qualms about the way they are trained. Two primary issues are that they aggressively scrape data from sources that explicitly do not consent to sharing their work and the data centers which have made hardware impossibly expensive for everyone but them and are so impactful to our underfunded electrical grids that they cause brown outs that keep people from heating their homes among other things, cause major spikes that cause house fires and destroy home appliances, and create an excuse to generate power using the worst polluting generators that we had finally gotten shut down to slow climate change (personally I'm tired of the wildfires and extreme storms from climate change).
So, I personally consider it extremely unethical to use the technology for that and several other reasons which are hard to touch on exhaustively in a post like this.
I tried, but didn't want to invest into it, so I used my 12-GB-vram gpu but no matter what I did it was worse than Lumo from proton,which is basically only one LLM from 3rd party that I am okay to use on occasions and since it has really good free tier I gave up on self hosted LLM.
With my NVidia Geforce 3070 card with 8GB of VRAM I can get 40 tokens per second on smallish models such as llama3.1 or qwen 3.8. I use it mostly for NLP processing of game server logs to create match summaries and I use whisper for audio-to-text transcription of videos.
While I do not have enough vram to put large logfiles into the LLM directly I was able to have it create a python script that generates a summary file at the end of each match. I feed the LLM the summary file and it generates color commentary that gets posted to Discord. Here is an example:
[🌟]Color Commentary: A Display of Skill on dod_napoli_b1! [📈]Multi-Player Action: On 08/05/2026 at 19:33:25, the action was intense on dod_napoli_b1, with key moments including the [Loft] capture by Inspector Lag (Allies) and [Axis Barn] capture by THE ROCK/BionicTwin (Allies). [🏆]Impact Players: Allies: Inspector Lag (Kills: 37), [415]Drago (Caps: 10). Axis: The Notorious C.U.M. (Kills: 39), [415]IronHeaD (Caps: 8).
$ollama list NAME ID SIZE MODIFIED
qwen3:8b 500a1f067a9f 5.2 GB 5 months ago
llama3.1:8b 46e0c10c039e 4.9 GB 5 months ago
$ollama run llama3.1:8b
Considering these factors and the specifications of your GPU, here are some popular AI model examples suitable for an NVIDIA GeForce RTX 3070 with 8 GB VRAM:
Computer Vision:
Natural Language Processing (NLP):
Generative Models:
Keep in mind that these models have varying computational requirements, memory footprints, and performance characteristics. You may need to experiment with different hyperparameters or architectures to find the best match for your specific needs.
To estimate which models will run smoothly on your hardware, consider the following:
For more detailed estimates and comparisons, I recommend exploring resources like:
Feel free to provide more specific requirements or task details if you'd like me to help with model selection!
total duration: 13.256527877s load duration: 71.80054ms prompt eval count: 466 token(s) prompt eval duration: 62.91049ms prompt eval rate: 7407.35 tokens/s eval count: 572 token(s) eval duration: 12.889617619s eval rate: 44.38 tokens/s
What game servers ? Have you looked into qwen3.5 9b ? Gemma e4b ? There is plenty of new small models after yours. There is r/lowendlocalai, a community of people with low computing resources but it is in reddit.
MiniCPM5 may be good.
Last night I managed to run Qwen Flash Next iq2 XS using Strata in docker on a VM
I passed through my 3090, 64GB of DDR4, 12 vcores of my 5960x,
Managed 90t/s
I'll probably try a larger version soon but QFN already seems a lot more capable than 3.6 35b A3B, and runs better than Qwen 3.8 dense in llama.cpp
I'm building toward exactly this, so take it with that bias. The cheap tier works if you lower expectations. An N150 mini PC with 16GB runs small models through Ollama fine, but 8B models are slow on that class of chip, and you'll feel it on anything chat-shaped. Where it shines is async work: summarize this, tag that, answer when it's ready. For phone access I'd keep it LAN-only and use WireGuard or Tailscale to get home rather than opening a port.
Disclosure: I'm working on a small open-source home box that bundles local AI with offline Wikipedia and media. What would make you trust a box like that over a setup you built yourself?
I've thought about bodging something together with n8n and Tasker so I can just screenshot something or send it a voice note and have it add an event on my calendar.
That being said my server is an 8th gen i3 w/ 12GB of RAM and I'm not willing to leave my desktop running 24/7 so idk if I can run a model good enough to handle this over there. Even if it is a supposedly menial "OCR and call tool" workflow.
Qwen 3.8 27B, running on a RTX 5090. Works so well.
i run k2 7b, it's delightful!
I run the AI service (OpenwebUi) myself, but buy the compute on OpenRouter. Self hosting models is not an economic viability for me.
Does OpenRouter have full access to your prompts? What is their privacy policy?
They have a flag Zero Data Retention that cones with the promise that after a call no data is kept on any servers. I'm using that.
It's contract level trust; I wouldn't bet my life on it, but overall it's a compromise I'm willing to make.
No, I prefer using my brain or other things that usually work 100x more efficient without hallucinations and don't required a shit ton of RAM that are way to expensive to buy right now.
Unless your brain is hallucinating this whole reality 😀
I wish
I write my own code, skid.
I self-host my own neural network, all within the confines of my own skull.
That 33% downtime though
Downtime is the friends made along the way
It's scheduled maintenance so it doesn't count toward downtime. Besides, I'm not trying to use it then anyway.
Scheduled most of the time, anyway.
It's better than github
Only 33% downtime? Damn, that must be nice. I really need to get a hardware update.
The downtime isn't the biggest issue, you can plan around that. The real problem is the random lag on some queries and volatile memory being lost (usually when there's a context change).
Me also, and I can barely afford to run that
How much RAM is it using?
Made me think of Johnny Mnemonic. https://www.youtube.com/watch?v=jS2GFqT5ycU
Iirc in a documentary, the brain runs on just 13w of electricity, so the more advanced and efficient AI gets, the less power and therefore resources it'll need. Maybe Johnny's world is just that advanced lol
I'm just glad that this is both Top & Hot comment. 🧠✊
I'm self-hosting immich (with ML-based facial recognition, semantic search and OCR) and psiTTSa, an offline text-to-speech engine.
(Prob not the answer you were looking for, but a reminder that "AI" is more than just LLMs.)
Does Immich sell/use/keep your ML-based FR? Or is this totally local a never leaves your instance?
That was always the question that bordered me... I run Leeche its a very simple and minimal photo album service and I really like it ! But my family want something that's more like immich but I'm very cautious about what i'm selfhosting and something that collects data points on my behalf is a no-go !
I do not run Immich but their privacy policy states:
https://immich.app/privacy-policy
Additionally in their docs I can find no mention of connecting to outside services. I do see references to downloading hugging face models.
https://docs.immich.app/features/facial-recognition#how-face-detection-works
Thank you very much ! I didn't expected someone doing my part to RTFM o_O !
Without doing again the heavy reading part, what should I expect from hugging face models?
If my reading skills are correct it should be local, but I'm really paranoid about this stuff ^^ Even though I read your posted links, I'm still not sure if I should trust them...
Something written in a manual and what the code realy does are sometimes far appart ^^'
It's local. You can tell by how your computer dies from inference, when you upload new assets 😅. And you can run the models on another computer separately and watch that one die then, when you upload new assets.
Immich is entirely free and open-source software and got a grant from the FUTO Foundation, which is fiercely pro privacy. As such, it does not engage in any data sharing. Development is financed through the grant and purchases of the server licence (I got one; it's ridiculously good software).
What’s your rig?
1070ti and some hexacore something cpu that's alright with around 24gb ddr4, I think one of the slower speeds. I also got a faster PC with another llama.cpp instance and I vibe coded a quick and dirty proxy which I can use to switch between the different backends. It's genuinely useful because I got code editors but also my Open-WebUI using it.
Sounds elaborate but generally it's all pretty primitive.
Also fyi long preloading steps because my vram is obv not good enough to hold the full model and kv cache and stuff, generation at about 15t/s tops.
My experience is right now we're at the stage where it's a good assistant but not great for actually taking over bigger tasks completely.
Am looking forward to:
Pretty much my setup as well. Though I'm using qwen3.8 27b-Q4_K_M under ollama. Everything running with docker. Add nginx and either a domain+port forwarding or zerotier/tailscale you've got an app you can get to from outside your house.
Right now not sure how I'm going to leverage it not probably to streamline my self hosted stuff, help with config mgmt and documentation, web page summarization, etc. we'll see how it does, just got it setup yesterday
In case you are unaware, friends don't let friends use ollama. The llama.cpp CLI has become pretty much equally easy to use, plus since you use Docker there's no need to compile anything.
Edit: typos and missing words
Oh shit I wasn't aware! Thanks for the heads up, I'll swap to llama.cpp
Oh damn what a useful link. Gonna check out those orchestrators, maybe I can replace my janky ass proxy.
Very similar, but I'm using Gemma4 12b. I could run 26b but that would exhaust my vram after a short interaction. And ollama. Open WebUI + Home Assistant hooked up with it, a couple very basic scripts I threw together, and connected to Kagi.
I'm trying to get my folks off the Gemini/Google bandwagon for the privacy aspect, but also because it's just cool from a tech geek perspective. For me, any questions throughout the day that I want info on, but not enough to do a deep-dive for an hour+, gets fed to Steve, my models name. Or I'll use it as a springboard and then properly read the citations for more info. And I feed it HA automations when I get stuck scratching my head on an edge case or whatever - it's alright but helps me explain what I want from the automation, and when it understands my goal, great, and when it doesn't I reflect on what I have setup and can usually go 'ah ha' and fix the issue. It's like a rubber ducky/pair programmer that you can both think is an idiot sometimes, but it doesn't judge you when you're the idiot.
That's a pretty good way to describe those kinds of models. Somewhere between rubber duck debugging and a really lazy intern. Better than an algorithm, faster than a human, but less trustworthy than an intern. It's great if you know how and when to use it. Maybe it'll become an intern soon enough, but for now let's focus on decompiling adobe into bankruptcy.
I've got similar running on my RX 7600 XT. It is not a great experience for interactive use, but fine for async tasks. I'm also running nemo-speak for live local dictation and meeting transcription.
Yeah running in the background type beat. Not gonna trust this with anything important but it's great to have someone just do shit for you or get a second opinion.
Yah doing similar on 5060ti.
That's gonna be quick af with the same models.
It is, i also use the gemma4b model on it for description generation for my immich photos and its super fast.
Random Question, I've been trying to offload some of my stuff handled by claud to basically this exact setup. However one of the things I manage has a pair of MCP servers that are required. I might not be smart enough, but how do you get Qwen to use an MCP?
MCP is a really dumb server that just takes requests and sends back data.
Somewhere in your frontend settings (Open-WebUI has it under "integrations" I believe, some backends can set these things too) you see a section called "MCP server". You need to keep your MCP server running somewhere (it's very slim, so not a huge deal usually) and then in those settings you put your IP and port for the server, for example "http://192.168.1.15:8080/". Usually if you did it right (sometimes you can check if it works in the settings), you can ask the model if it has access to the tools and it will tell you.
What's happening in the background is that every conversation the AI does, it gets a short preamble with some context about what it can do, and if you connect web search, custom skills or an MCP server (it's just skills but they are run through their own server basically), the AI sees all that in this preamble if it's set up correctly. And then when it feels like it needs to use those tools, it will check the description on how to do that, and then it uses them.
Thank you, I'll give this a go
Running Qwen 3.8 27B dense on some R9700s, and Qwen 3.8 flash next on dgx sparks. I use it for code gen outside work, including an automated Dev pipeline that rubs on its own. When I did the math comparing to token API pricing of Opus 4.8, it covered its own costs in a few months. With the sparks I get about 40tg/s, 1500pp/s, but can get concurrency up to 5 stably. With the R9700s, I get ~100tg/s, 3000pp/s and can also get up to 5 concurrency.
I access it over wireguard to the network that it is on, and use termux to ssh into a build machine for dev over CLI using https://pi.dev/
I also started building https://fcli.dev/ to make integrations easier; the idea being that LLMs should just implement simple reusable functions instead of repeatedly making complex curl requests to integrate with a service.
I also use LLMs heavily for work; Qwen 3.8 seems almost on par to Opus 4.8 IMO. I also don't have to worry about token costs, and it often responds faster than Claude would because there usually isn't a queue on my machines. At this point, I wouldn't complain if my work provided me 2x B6000 instead of a Claude subscription; I'd still be able to get all my work done.
Would be interested to know your experience with fcli if you’d share more. Just hearing about it.
I mean, I haven't really started discussing it outside IRL friends, so not surprised you haven't heard of it 😅
In essence, the problem is twofold:
So fcli is intended to be an easily extended, configured, and authenticated CLI. The "one CLI" to do everything your LLM needs. Ideally, you set up a personal repo to store 'ucks' (f packages) in, and you set up credentials for that repo so your LLM can push any new ucks to it. You can then share that repo with other users if they could also benefit from any of that functionality.
As you go along in a session, when you notice the LLM repeatedly running similar verbose commands, you can have it create an uck for it instead and save those tokens.
Likewise, if, say, an MCP server doesn't have full functionality, you can have your LLM import the SDK from npm into an uck and write a basic fcli wrapper around it to start using.
The other nice thing is that it makes your tool configuration repeatable between LLMs and harnesses. You don't need to set up your MCP servers in both Claude and Codex and Pi, just in f and tell them to use f.
There's also a creds uck to make your secrets more portable and (a bit) safer.
In general, I'm trying to model it a bit like pi.dev, but as part of the tooling instead of the harness itself. I think they work quite well together :)
Interesting paper comparing MCP and CLI
Can you elaborate on LLMs using CLI more effectively than MCP? Do you have any sense of how likely it is just your domain or the MCPs you're using? Do you have any theories about why that may be?
MCP in general:
CLI in comparison:
CLI does of course have its own downsides, I think the biggest being security. Give an LLM an MCP server, and all it can do is what that MCP server supports. Give an LLM a Turing complete shell, and it can do anything.
MCP also tends to be harder to set up and harder still to fix when it doesn't work. I've watched Claude fall back to curl commands when a connector doesn't work many times. I've never seen it try curl commands, fail, and then try to set up MCP.
I think this is all fairly domain dependent, but I think my domain (software development and general agenetic assistance) is very common. If you're using an LLM in a product that you're selling with a fixed range of features, MCP is probably the way to go. If you're using an LLM for dynamic functionality with expanding capabilities, I think CLI might fare better.
My main reasoning for all of this is flexibility. In the history of computing, a program has always had a fixed output for a fixed input. LLMs are at the point where they can handle what is effectively an arbitrary input and give an arbitrary output. Giving an LLM a fixed set of tools negates that newfound ability to process the unknown. It's like giving someone a sledgehammer to bust rocks, but tiring one hand behind their back; the work might get done, but not as efficiently.
I agree with you for common CLIs that the model will natural have in its training set, but when building your own CLI it seems like the self-documenting nature and progressive disclosure of MCP would have a huge advantage.
With a cli you don't have the prefix cost, but your agent also doesn't know that the tool exists, what it's for, how to use it, etc. once you give it that information, then you have the same issues with prefix cost, right? Changing tools does break the prefix cache, but changing tools is a thing you can do. You can't change (model awareness of) CLI, and if you could then it'd also break the prefix cache, or you'd have tool instructions scattered throughout the context without structure.
Dumping to file is a harness thing, not a MCP thing. A harness will do the same with large terminal outputs.
Likewise setting up a new CLI can be complicated. I never see an agent try to install an mcp, but I also don't typically see it try to install a new CLI.
I don't see how mcp is any more a fixed set of tools than CLI is. It's just presented to the agent in a structured way, which is important for discovery and instruction following.
But yeah ease of chaining is certainly useful wrt CLI. That's a huge win for sure.
What a timely comment.
I recently gave my kid a gaming PC for his birthday...which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB
Combined with a 5600x3d and 32gb of system memory.
I just started messing with local LLMs yesterday. "Getting started" is rather easy.
pacman -S lm-studio(I use arch, btw), launch it, download a model that your hardware supports, and start talking.Right now I'm working on getting it to talk to vscode.
I used Cline as my VSCode chat plugin for models in LMStudio. Worked pretty well but I have moved my local LLM stack to Hermes in the meantime.
I tried using zoo with qwen3.6 and 2.5 but it failed horrendously. My card could only handle like a 16k context window with those models, and the prompt alone was like 11k.
Tried opencode and that was fast...but can't get it to interact with system yet. More playing to do.
Not only do I run and share it. I've built an entire service to allow everyone else to do so safely. https://aihorde.net/
Yes, were all on c/LocalLLaMa
I'm running qwen3.8-next-flash on a rx7900xtx plus 128gb of ram. It serves my hermes agent. I direct it through matrix. I also forward an endpoint to my VPS to use for anything. Mostly opencode.
Is that the only community? I see the last activity was 2 months ago.
that means no one from your instance is subscribed, if you subscribe then you'll get new posts
you can see that there are new posts here https://sh.itjust.works/c/localllama
Looks like Reddit.online blocks a lot.
Been subscribed. Still shows two months ago
Posts from 16 minutes and 2 days ago here. Maybe your instance is not federating with theirs.
oh yea feddit blocks shitjustworks https://feddit.online/instance/sh.itjust.works/posts
maybe @[email protected] could ask their admins to convert the defederation to a "silenced" or whatever that feature is on PieFed
Replaced to other on top but look at feddit.online vs Lemmy sh.itjust.works. I think I’m going back to lemmy
Thank you for confirming. I didn’t know I was even missing out on posts. Your follow up, I don’t even understand. Maybe I should go back to Lemmy (I think my account is on sh.itjust.works) and do my own filtering.
PieFed allows admins different levels of blocking instances, they can do defederation but there are levels softer than that
Are you saying that the admins of feddit.online can do that only to my account or is this a instance as a whole?
Yeah default blocks or silencing or something like that
https://piefed.social/comment/13343280
I think that’s it. Even though I’ve been on Lemmy then PieFed for over a year. I still end up not understanding any of this but I think this is one of my core issues here. I would have never know.
I wish I could. I don't have the hardware for it.
I run Gemma-4 4B.
It's not the best model, but it runs with CPU only and with 6 ~ 9 G of RAM.
We did the maths. Didn't seem worthwhile and the payback period was a very long time
I think the value is in the privacy. You can hook it up to knowledge and capabilities you'd never share with a cloud AI. But if your use case is strictly coding, I agree.
True but if you happen to buy broken macs you can just take thier board with all the unified ram it's become quite the thing recently even Asus duo with 32gb Intel 9 core ultra board's have been started to be repurposed like mini pcs for ai use. For about 1000usd get the same level of intelligence as o1 high with gemma 26b I think with enough left over for kv cache.
24GB GPU
Qwen 3.8 ( 2 variants )
Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab
Zoo Code>lama.cpp for coding , mostly scripts and html
Qwen 3.8 was a game changer for me.
What variants? And why two?
For coding with large context: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp Anything else: Qwen3.8-27B-GGUF:Q5_K_M
Yes i do.
I esperiment a lot, but my stable setup is Qwen 3.8 27B Q6_K on my home server. Hardware is Ryzen 9 5950x with 64gb ddr4 and two old but effective NVIDIA RTX A4000 for a total of 32gb VRAM.
Qwen 3.8 is the best model I can run in terms of output quality. It can perform all tasks pretty well and it's reasonably fast at 500ts prefill and 30/40ts processing. There are much faster models but honestly not as good.
I can also run gemma4 all on CPU, with no GPU at all.
The key point is having both GPU on pci-e CPU, not on chip set bridge, so I am using a PCI riser 8x8x to share the 16x slot between the two GPUs. Inference speed is unaffected, but prefill benefits immensely.
Software side, use llama.cpp on Linux, you get the best performance possible at the expense of a bit more complexity compared to other engines.
From a power consumption perspective, I tined down my CPU governor and maximum frequency, and also the GPUs maximum watts. At ide the system is around 80w. At peak, it's about 280w with the limiters, and inference speed doesn't suffer too much. Without limiters I can easily reach 650w.
Edit: I run llama-swap in front of llama-cpp, and openwebui as agent manager/ web GUI. I also have ComfyUI installed because why not but seldomly use it.
You can read more of my setup on my wiki https://wiki.gardiol.org/8-llm-and-ai (no ads, no monetization whatsoever because fuck a hobby should not be a side gig)
Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track
I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system
I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents.
Glm flash (4.something?) does pretty good as well.
I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol
I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in.
One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources.
Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up.
I think it’s cool as fuck. I geek out every time that it works.
Oh, forgot to mention I am running it on a Mac studio with 96 gig of unified memory. I think it’s an M3 ultra chip.
Do what makes you happy mate! Tell the naysayers to fk off.
@aurion The hybrid split sounds right. For us, a local 9B on CPU took about 40 seconds just to read a 2.7k-token agent prompt, then generated at roughly 3.5 to 6.7 tokens per second, so prompt size matters more than people expect. Is most of the wait on reading the prompt or on generation for the Hermes agent?
Hey there! Sorry for the delay, I didn’t forget about you.
I didn’t know how to answer your questions, but they are very good questions that I wanted to know myself.
I’ve been working on a testing plan to figure this out. But life is got super busy and it’s been hard to focus on it.
Hopefully I’ll complete it before the weekend is over and will share the results.
Not me. Call me old-fashioned, but I prefer organic.
What does it do? I thought about trying this but I literally never have a need to generate text.
I’d like to, as I currently drop $20/month on Claude without perhaps using it to the extent that my money would let me, but what’s “good” that can run on a 16GB RAM, i5 12th gen, 3070-equipped desktop?
Gemma 4 12b unified as a q6 gguf fits with about 24k context inside your vram. koboldcpp+sillytavern works well on the software side.
Sc117
Might wanna try a ternary model, which are a lot denser in comparison.
Ternary Bonsai 27b is 7.17GB in size, which might fit on your 3070 depending on how much free VRAM you have (and with CPU offloading it should work in all cases). You can also try their edge-computing model designed to be working on smart phones here (iPhones seems to have a limit of 6Gb RAM max for such things) - that one is less than 4GB in size, which should fit in any case. In my experience they do work very well compared to other models in that size class.
Be aware that they need a specific llama.cpp build found here
This is the question to spend your Claude tokens on. It helped me tweak my setup to get better performance.
I have a Qwen model running on a Mac Mini with Ollama - it’s reasonable for Home Assistant, responding to iMessages and similar.
Yes, I installed an LLM at home. I use it for many useful things:
I have a RX9070XT and llama-swap to switch models on the fly. I'm thinking about installing a second RX9070XT so that I don't take away the translation capability from my lemmy instance while I stress the other neural network with other tasks.
Hardware:
2 x ASUS GX10 128GB(each)
2 x Mac Mini M4 Pro 24GB(each)
The GX10s are clustered and run DeepSeek-V4-Flash-0731-Abliterated. The M4 Pros run various utility LLMs such as Qwen 3.8
Software:
OpenWebUI
Hermes Agent
LiteLLM
Llama.cpp
vLLM
StoryBinder - custom web app that I use for story/script writing when in the mood
LASM - Local AI Stack Manger - custom web interface to manage pretty much all aspects of my local stack. Agents run on the 4 nodes as well as my NAS(Download GGUF/HF snapshots once and copy out to the nodes)
Have yet to connect my Home Assistant into the stack.
For mobile access I use Conduit https://github.com/cogwheel0/conduit
I find the idea interesting. However I wouldn't get the commercial service you are advertising. Maybe something that can run on an older CPU as well. Doesn't need to be fast, but if it uses up the idle compute I have on my rented server, then why not.
I'm not advertising any service. I added an image because Lemmy wouldn't let me post the question without one. I think I may have chosen the wrong post type or something. Sorry for the confusion.
No worries. Yeah, the user interface is difficult. Thanks anyways for the post, it motivated me to self host my own agent, and it works quite well actually.
This is something I will do probably within the year, but it feels to me like the area is still moving quite rapidly. The model will naturally depend a bit on the hardware. I'm looking at used AI cards because it looks like currently for about $2k you can get one that will run ~350B models, and GLM is nice in that space, though I will experiment with other models once I can run them.
Things I will do:
I have a ton of ideas for conveniences AI could do if it had total privacy and security. In short, orchestrate all of the tools and knowledge that I already rely on constantly. Integrate calendar and private info like SSNs, VINs, who has what doctors for what, or who was sick when, smart alerts like the overnight temp will be below freezing next week so make sure the house is winterized, or I wanted to see this movie and it's releasing or will be leaving theaters soon — oh and next weekend you're visiting the grandkid, so it's either now or I'll remind you again when it comes out on streaming.
Just having a personal knowledge base always on and totally private would be handy. It would be like having a personal assistant maybe concierge if it could be trusted to that degree.
Your use cases are quite similar to what I've been trying to action.
One question I keep wondering about is the "security" aspect of it. And how the inputs to the model might work. Yes it can read your calendar and you could "Jeeves, please remind me to hang out with Wooster next week" but it's easy to forget and I have a hard time logging spontaneous events.
Any ideas on how you imagine the friend/activity tracker could work?
Was thinking maybe it could read my smartwatch location history. Or scan the photos I take. But that gets dystopian fast (yes it'll be local but then I'm worried about security).
Fun to ponder!
I think having a chat interface is part of it so it's friendly and convenient to use. I think someone suggested a matrix bot and that would continue the theme of privacy. You could have different channels or groups for different personas. Maybe one is your Majordomo, another is a simpler calendar bot, another is for journaling, etc.
Then, wire in some cron jobs to invoke the AI to prompt you on various subjects. "Anything you want to share?" every day at 9pm. If you miss a day or two because you were busy or asleep, you can catch it up later or not at all. You could even have it ask what you did at work if your life is like mine and they kinda blend together and you don't know how many hours you worked.
Come to think of it you could use it as a note taker. It could separate work notes from personal notes.
I self host both at home and at work.
At work (for completion and generation) we have a now quite old A6000 running mainly Qwen 3.6 (though I'm trialling 3.8 from Monday) with OpenClaw as the harness. The use case is mainly triage and pre-work on bugs. We have an agent which can be given an exception report with a bunch of information like stack trace and local variable values and it will go off and gather the required code, order files, and data to analyse the issue and come up with a fair idea of the cause and possible solution. This saves devs and support team time. Also looking at adding a PA agent to this which will be able to read emails and calendars and be able to set up (with approval) meetings and move things around and generally make someone's life a bit less admin-y without having to hire a full PA role.
Also technically in the AI space we have a fleet of Pis with Hailo AI hats running object and anomaly detection in visible and LWIR spectrums for QC
Are you planning on using a specific package for your PA agent? Or is this something you plan on developing in house?
Using openclaw for the harness and then attaching office 365 mcp with an Entra ID App token to give it access to email, calendar, and teams. The MCP format means that we can explicitly control what it reads and writes
If you haven't seen it already, take a look at ![email protected]
90€/mo to self-host something? Lol
That infographic looks to be for a VPS deployment, which is where the cost would be coming from. They're comparing to a $9,000+ OpenAI payment.
I'm new to Lemmy so when I tried to post something it said I need an image so I just picked a random one which looked vaguely like self hosting AI
You don't need an image, and the one you picked makes your post look like an advertisement. You can edit your post to remove it.
I can't remove it because there is no delete image option (in my client at least; I am using Voyager), only a replace image. I suppose I could replace it with a single pixel image but I think at this point it doesn't matter.
Yes, I have a desktop with a radeon 9700 ai pro with 32GB of VRAM. Its more than doubled since I bought it but its basically a down clocked 9070xt with double the VRAM. I also have a little arc b50 in my server.
Mostly yeah use it for privacy reasons. I run llama.CPP directly and access via open webui. I host a searx instance for it to use as web search. Mostly run qwen3.6/3.8 or gemma4 on the radeon and almost exclusively gptoss 20b on the B50 along with a second small model for tags and titles in open webui. Yeah I just use it mostly as a basic chatbot or as an agent in obsidian using the engram plugin. Helps search and summarize years of notes and works pretty well.
I've tried to use it as a coding assistant in zed without much luck so usually just post my code into open webui for feedback and it works alot better for me.
Yeah I can access it with other machines on my network including phone with conduit app. I can get to it on the go by connecting to home network with wireguard and sending a magic packet to wakeup desktop.
I havent done any image gen nor have much use for it outside of maybe using it to make app icons in the style of papirus icons for programs they don't and won't support.
It definitely feels more comprable to first ai wave than current frontier models but the privacy and price trade off is worth it to me. I'd say using it in obsidian has been the best use of it so far and one I certainly wouldn't trust to an external AI.
I run lemonade on my strix halo box, I change the models I use, but the qwens are pretty good.
I access it via Tailscale on my phone, then I have a page I can use. The Lemonade interface forces you into the app, so you’ll want some other interface for your LLMs, as the app is junk.
I used to run llama.cpp and llama swap, but lemonade combines them and makes managing different ai backends easier.
I've been eyeing lemonade as I am on llama and swap my self. How much of a pita was the migration. I'm a lazy bastard.
Lemonade is pretty good, especially for AMD GPUs and the lazy, but it's pretty opinionated (like ollama but not as toxic) and it's not easy to directly control llama-server parameters. It feels a bit commercial / we want to IPO.
Seeing as you already have your parameters, llama-swap might be a better fit, lets you configure llama-server directly in a yaml, and switch between them. You can even have multiple versions of the same model (e.g. thinking or not, try out different parameters) and change without reloading.
Both work fine with e.g. Open WebUI and Pi (lemonade its own integration and llama-server needs an extension).
If you update the model json you can pass through almost any parameters you want
It is an extra step though
Good to know, it's been a while since I played with it and the docs were sparse.
I pointed lemonade at my llama.cpp model folder but it uses json files to configure them. I think it was fine for importing them or you could have an agent write them for you. After that you can manage models inside the app
Overall it wasn’t a huge deal, it’s easier than managing llama.cpp directly with rocm updates.
I recently gave Strata a shot for my local AI. I'm very impressed with the performance they have eeked out of my hardware. I've got a rather beefy gaming desktop I built as an AI server back when the current craze started, before prices started going up, but models up until this point really failed to take full advantage of that hardware. Strata (which uses a Qwen model) both simplified the setup process, and it maxes out the hardware when running.
Strata acts like an OpenAI compatible API server, I then have Hermes as a "harness" to make an "agent", which I interact with on "buzz", which is like discord, but made for bots. Although Hermes also supports discord and several other chat platforms.
If you have the hardware to run Strata, I recommend setting it up, it comes with a basic chat interface, which you can then grill to assist you with setting up Hermes and whatever else.
One thing I didn't mention, is security. I highly recommend you treat AI as unpredictable, unreliable, and capable of being taken over by bad actors. This isn't a problem for a pure chat interface, but Hermes gives the AI tools to do things, and those tools could be used to take over your computer, scan your network, and create all kinds of havoc. For my part, Hermes is running in a virtual machine which is on an isolated network. I keep tight control on what it does and doesn't have access to in my network main network. That may not be something you have the hardware or knowledge to do, at the very least I recommend running Hermes in a docker container, and not giving it access to your personal desktop nor credentials to things like your email account.
Three models my self, qwen 27b as a dev assistant, a small model rigged up to handle voice input as an assistant that's connected to my intercom system though out the house. Mostly to run my lights/music/audiobook play back. It's fun having jarvis control the house after all. Then a image generator.
Llama and swap on a 7900xtx.
I haven't gotten around to setting up phone access but wireguard for remote access form work in my laptop.
Honestly it's all basically over priced toys. I'm baffled that people rely on this stuff. I look at it the same way people do with model airplanes or RC cars.
Over priced toys that are hell of a lot of fun. Break frequently and require more care then they are probably worth to use safely lol
Sure. I have Home Assistant with STT and TTS. And I use KoboldCPP to experiment with smaller LLMs. Nothing fancy, it'll run on my old laptop, albeit very slow.
Yeah, I have a fairly useable setup although it's only accessible on my lan. I have a web server but it doesn't host the llm as it's not powerful enough.
Rx7600 8gb vram GPU 5700x CPU 64gb 3600mts ram
Hosting ornith 35b a3b iq4_xs Llama.cpp server Pi harness or llama.cpp webui Searxng web search Coding sandbox Restricted filesystem access
Get around 40 tok / sec and it's really useable. Not as good as Claude sonnet, but good enough for most things.
You can setup a reverse proxy on the web server to direct a subdomain over to the ai server if you'd like to have it accessible outside your lan
Yeah, I know how to do that very easily, but I don't want to have my pc running 24/7 just so I can use the llm when I'm out of the house. My heaviest use is at home. It's a large power requirement for minimal use.
Hell yeah. I totally agree.
Yes, with OpenClaw I can expose my agents via discord apps on my home discord server, I like that route because of the rich media and UI options (agents can give me drop down selector lists and approve/reject buttons). I have two 24gb Nvidia cards for local models usually Qwen, Mistral, and nemo based models for different purposes. I can chat with the agents anywhere I'm logged into discord and I expose privileged control surfaces and nodes via tailscale and family/shared endpoints via caddy and cloudflared
What are the agents doing for you?
I'm exhausted from writing poetry all the time I want something that can do that for me.
So one thing that's really helpful to automate is checking all the repos for my self hosted stacks and testing/validating/deploying new versions when they come out. They're also good at writing helper scripts in Python to automate things deterministically so I don't have to keep using LLMs for it. There's an agent that is focused on learning and managing my wife's school schedule, ingests her grades and syllabi and reviews her homework and makes helpful 10 minute podcasts on concepts she's struggling with in her engineering program (not notebookLM, a self hosted lookalike), there's a domestic agent that accepts uploads of grocery receipts and does our meal planning and schedules time for meal prep and chores and divides it up between us and the kids depending on everyone's calendars, there's a home media agent that manages qbittorrent and my ratios on private torrent trackers and files new manga or TV shows or movies ebooks audiobooks etc in the right library when new things come out we might like, viewing/reading/watching metadata tells the agent what content is popular to queue up in the next sweep - there's a home assistant front door agent that can make automations or report sensor state or trigger scenes or entities in the general text channel of our home discord server and can tell me where my phone or keys are thanks to ESPresence being able to use cheap ATOM esp boards throughout the house (tiny little USB powered things with Bluetooth and wifi) to triangulate the location of any watched Bluetooth device or tracker tag in 3 dimensions through floors and walls - what else..there's a home security red-team agent that looks for security holes in my cloudflare tunnel/kv worker/tailscale/caddy/protonVPN routes and monitors a cyber security RSS feed, I have a job/market research bot that ingests another RSS feed for news relevant to my industry and organization and gives me email and voice briefings on new regulations and corporate leadership signals, stuff like that. What else.. there's a game dev bot where my kids can talk to a discord bot that routes to a game dev agent that has Godot MCP and Unity MCP and renpy MCP and lots of other fun tools like Roblox MCP and Minecraft to make custom games we can deploy on Google cloud for their friends to play (my daughter is really into visual novels for some reason I don't get it) - all sorts of fun stuff. Discord is kind of the family shared interface and then the privileged control planes are routed to my phone through tailscale and the home media server stacks are exposed through cloudflared.
I'm confused about some of these. For example everything you mentioned in the torrent agent (maintaining ratios, watching for new media, seeing what's popular) are already settings in the *arr suite that uses significantly fewer resources, no?
This is all basic functionality of Home Assistant, yes? I'm confused about what is the agent is doing?
In the case of HA, the agent is receiving ambiguous natural language requests and converting them into HA MCP calls. I can open the HA app and press buttons, sure, but the agent can reason over time about patterns of water/electricity use, provide natural language feedback, push back on my automation ideas with counterexamples from sensor logs - and all the sensors in the home and on the phones carrying the companion app are rich signal sources for an agent. It unlocks neat tricks like when our babysitter comes over the front door agent can alert her if there's a water leak and walk her through shutting off the water to the house or locate the backup key for her. It can DM me when someone rings the doorbell and carry on a different conversation with STT depending on if it is a person wearing a uniform or holding a clipboard for example thanks to Frigate and some custom vision ingestion with Qwen 3 VL.
Regarding the torrents, that agent keeps track of my ratios on the private trackers and I can give it natural language feedback about what I did or didn't like about a show or movie and it will suggest alternatives, find them on a private tracker, download them, unpack them, and ingest them in the library for me, keeps track of the families like and dislikes and suggests media to watch together, watch private trackers for open invites or interview slots and take the knowledge interviews for me on IRC, deduplicate downloads and hunt for releases with minimum quality standards etc. one of my first tasks for it was replacing my music collection that got damaged when my external hard drive got dropped with no backup and I had to get it extracted in a clean room. Lots of rare lifetime accumulated mp3s gone - mostly recovered now. Searching for replacements would have been a long slog so I automated it.
To your point some of these things could be done by deterministic python scripts and in fact the agents write those to save inference costs later and most of this stuff I could do by siting on my computer an clicking on stuff and reading - the point is that the agents constitute a natural language control surface to trigger all of those actions for me automatically so I can just send a couple chats to a discord bot and it will be done a few minutes later. I'm a busy person I dunno if you can relate. Busy career, got a family to support, parenting obligations, no time to hang out playing video games any more sadly. Building stuff takes time and effort and you have to learn and read about how to do it or pay someone else to do it for you so I was stuck on the easy convenience of windows and Netflix and Amazon/kindle etc. Not anymore! It started with having to install docker desktop to get OpenClaw working and then pretty fast I figured out pretty much everything I wanted to do now that was not work related is better done on Linux and better done self hosted. I would never have gotten over that hump without LLMs.
I have an LLM running object detection on a b580 that’s also handling camera rotation in frigate. It’s hanging out 95-120 watts
I am running deepseek-r1:1.5b on a laughably old dell laptop (circa 2011) cpu only (no gpu) and 16 GB ram. It is slow but works as a general assistant for Newelle. Mostly proof of concept but so far it has helped me proof read documents and tweak its own setup. It al also runs clean up and organization jobs on my laptop to manage the mess of old files on it.
I run Gemma4 12B on my RX9070 XT. it's not permanently on, I just turn it on when I want to use it. Access it through a browser. Does what I want out of it. Nothing "agentic".
I don't want to create an extra post, so I ask here.
Where do you guys get your hardware from, for an affordable price?
I'd like to run at least a 27b model but my hardware is too weak. I'm on a 16GB unified mem, M4. Tried running Bonsai 27b (it's Qwen architecture) but I had to close all other apps and it is not in a state that I could work with it.
A Tesla V100 is fairly affordable but older and won't run newer quants like FP8 or attention kernel routines. But it will run a pile of models, even most new models are compatible or usable.
I will ask Santa Claus for that GPU <3
Wonder if anyone has looked at power costs of running local ai. Some things sound nifty, but if it's costing me 20 a month it doesn't seem worth it
Yes, I run Ollama on a NVIDIA 1080 Ti at home. I run granite4:3b for my Home Assistant assistant for my voice assistant, paperless-ngx, OpenWebUI for chat, and a few other things. For speech to text, I run a custom-built hybrid speech-to-phrase + parakeetTTS (requests that match speech-to-phrase take priority over parakeet.) I run GOModel to route requests to my local Ollama instance, or DeepInfra (cloud-based for larger models), or Tinfoil (cloud-based for encrypted LLM chats.)
As much as I love to use local-everything, only my smarthome+basic chats go local. Trying to fit both the STT and LLM models on the same GPU at the same time while also using this computer as a media console is challenging. I've recently started looking at the Hailo m.2 accelerator board to see if I can shift the STT to it.
Tried running a decent Qwen on a 3060 TI but I only have 16g ram so it kept getting stuck and not completing tasks.
Which Qwen?
Ollama container, 2070 super, 3-7b, meta and qwen
I use it to ingest tons of news rss, find dupes, summarize, and then post that output to mesh a mesh network.
I also use it for a self hosted mealie to figure out recipe->ingredient.
I also use it for receipts. I translate pdf -> txt, feed the text to qwen, ask it who is the vendor, what is the price, what it a reasonable memo for the receipt, and give a dozen categories, which is best.
I am actually very interested in the news feed thing. How do you find duplicates in the news feeds? Wouldn't you have to pull the actual articles and then have MB of data to shift through which would exceed the window? Do you use a vector DB? Do you just have it use curl or do you pay for an Internet API service?
Pardon the mess, this turned out to be quite a lot of text.
No vector DB, no paid API, no full articles. Plain Python pulling free RSS feeds, plus Ollama running on my own 2070 Super. RSS already gives you the headline and a short blurb, so there are never megabytes to sift through. Every Ollama call is one story at a time, so the context window isn't an issue. Llama3.1:8b for summaries and scoring. Gemma4:e4b for clickbate work, nomic-embed-text for headline duplicate vectors.
Find duplicates:
Python: strip each headline down to keywords (drop words like "the", "says", "update") and only compare headlines that share an uncommon word. Two headlines match if they share enough keywords.
Python: sends batches of 128 headlines to Ollama
Ollama: turns each headline into a vector using nomic-embed-text
Python: compares the vectors, and anything over the config threshold counts as the same story. The vectors just live in memory for the run, no database needed. If Ollama is down, the keyword match still works on its own.
Regen short summaries:
Ollama: Generate a 2 sentence summary from the feed text if available
Get rid of clickbait:
Ollama: Detect clickbait from the headline only. Classify this news headline. "informative": it states the news itself (who did what, what happened, what was found). Dramatic or emotional wording is fine if the fact is stated. "teaser": it withholds the key fact to make you click: e.g. "The Surprising Reason...", "The Class That...", "Why X...", "How X became...", "What we know about...", "Is X about to...?", "Then things got weird."
Unwrap clickbait and rewrite the headline if possible, otherwise drop the story
Ollama: A clickbait headline hides a key fact. Question it leaves open: {withheld}
Does the text below explicitly answer that exact question? Mentioning the topi talking around it, or answering a different question does not count.
"answer": the answer quoted or closely paraphrased from the text, or "" if the text doesn't give it. "headline": if answered, a plain, factual replacement headline (at most 90 cha states that answer, using only facts stated in the text. Keep the text's certainty: ...
Respond only with JSON: {"answer": "", "headline": ""}
Headline: {title} Text: {text}
Score each headline
Ollama: return a summary and an importance score in the same call:
SCALE = ("1-3 niche, local or trivial; 4-6 notable; 7-8 major national or inte "9-10 historic events or emergencies affecting millions right now.")
Python: rank = importance + a bonus for more outlets carrying it - points for age
Don't put US articles in World unless they have non-US countries involved
Ollama: Which countries is this news story about? List the country where it takes place and every other country that is a real party to it (acting, targeted, negotiating, fighting, trading, sanctioned). Use common English country names, e.g. "United States", "United Kingdom". Ignore countries only mentioned in passing or as background. Always name at least one country: if the text doesn't say wh give the country where it most likely takes place.
Respond only with JSON: {"countries": ["..."]}
Headline: {title} Text: {text}
Python: if the only country is the US, it's left out of World
Everything gets cached per story, so each one only goes through Ollama once.
Wow, this is an absolutely wonderful write-up, thank you!
I've been using NixOS/nix containers to host some llama.cpp servers and Hermes Agent. Though I wrote a flake to get Project Maya working so I can serve GLM-5.3-Flash and in legit blown away by how well it works. It's a fork of strata/llama.cpp that focuses on where the relevant experts are loaded and has a neat dashboard to track their use. https://github.com/mw00/project-maya
My server is assembled into an openair mining rig with a Tesla P40, RTX 3080, and a few Tesla V100s. All of which I bought used off eBay.
I purchased my first domain to self host a matrix server that I can access via my phone and I also have tailscale set up so I can ssh into the server via my laptop while traveling
"Some practical notes"
Looks inside
Three slop in a trenchcoat
I just got qwn3-4b-instruct working on an Intel A750 with 8 gigs of vram.
I use it to power Home Assostant/Voice Assistant on a few M5Stack echos around the house to replace cloud based smart speakers.
I use llama-swap with a couple of different models, one 8B as a general chat bot, with offloading to several MoE-models for specific workloads. It's all run on an old dell xps with 64GB ram.
Mine is used for code reviews, general questions, coding assistance and for anything else I can think of.
it also doubles as my media ripper. I initially built it specifically for that before AI was even a thing.
I run too many models to count really. everything through CLI because I'm dissatisfied with all the sloppy UIs.
Look at the billionaire flaunting their wealth
in my defense I bought the majority of that in 2024 for around $300.
I was surprised to see it's currently listed at over $2000.
Ollama, openwebui, running the ~20GB Qwen models on a 3090
I wouldn’t say I ‘run’ it, but I mess around with it
I have a base level Mac Mini I got before the price increase a few years ago, also from when models were smaller and my ambitions were less. I’ve contemplated upgrading it, but I have other things to spend my money on.
M4 processor, 16 gb ram, 256 gb storage, and a 4tb external HDD drive (an old NAS drive).
It’s fine. It’s mostly a docker machine, but I do throw AI tasks at it a few times a week. Mostly stuff that I just let it chew on for 5-10 minutes and come back to.
I run openwebui, and have several models I bounce between. I mostly run Gemma4 (Google) and Granite4.2:8B (IBM).
Editing to add: the self hosted version of all these models, via llama, just in case that was unclear.
Hardware is a 5070 12GB VRAM, an AMD 5950x and 64G of DDR4 running on Nobara (Fedora spinoff).
I use 3 models for most things: Ternary Bonsai 27b and Gemma 4 E4B Uncensored Q8_0 for everyday stuff, and Qwen Agentworld 35B A3B Q8_0 with CPU offloading for tasks that require to keep the model on track for long periods of time.
Everyday performance is best with Bonsai for me - it comfortably fits in my VRAM and is noticeably better with staying on task than similarily sized models. But if i want it to work overnight on a task without issue, the lower speed of the agentic Qwen is outweighed by its capability to do this.
They are allowed tool calls for websearch and reading webpages and have a sandboxed directory for R/W operations, but anything regarding command line and R/W access in other areas has to be manually authorized by me (i'm not willing to play with fire here).
They are currently not reachable from outside, but i am currently building a smaller machine as a home server, which will then be using a 3070Ti and 32G RAM to provide probably the Bonsai Model for me on the go per VPN.
The smaller models are here for quickly creating documents i do not care to do myself, RAG for technical documentation, creating quick python scripts for automating small tasks and teaching me RegEx and Python(mostly by analyzing the LLM output and playing around with it) and also some RP as NPC in a fantasy setting. The agentic model is currently working on cataloguing my games collection (around 36000 titles) by looking up games, their genres and descriptions and filling in blanks in the database. (Funnily, this task was refused by any model freely available because of the size of the database. Even the local models moan and bitch that it takes so long. No shit, this is why i outsource this!)
Framework desktop
As a service? No, not really.
I use llms for specific tasks, usually small models, for things like identification and tagging (immich, paperless), translation or transcription (whisper), etc. I have a bunch of larger models on a Mac mini and my work laptop for various testing purposes, and have used them to make summaries and other such things.
They aren't great for search IMO, and search summaries can be quite bad.
I do find them useful to interrogate an unknown rs232 protocol or similar, though, which is where larger models come in handy.
Corporate LLMs are generally garbage to me not only because they are just data farming, but also because they really aren't good at everything. They might claim it, but the reality is a tailored model is often the better approach.
Using openwebui as a front end for llama.cpp is a decent strategy, combined with sub-agent routing (so generic model hands off to tailored models then returns the right info). This can be useful for some scenarios.
A caveat on coding - IMO, they are mostly bad at it. If you are using it to generate a small part of something that needs to be repeated they are handy, but at scale they aren't great, meaning for the entire project (or even an entire plugin or module, depending on complexity). Maintainability becomes a serious issue. While they have improved somewhat, there are limits to what they can do.
Macs with M series chips are good options for a hardware base since the memory is shared, and IMO is one of the most cost effective options over a dgpu with a ton of memory. A dgpu with a decent bit of memory (say 8-12GB) is a solid choice for a model thats larger (27b with 3 or 4b active as a mixture-of-experts model) especially if you have an iGPU for regular graphics tasks. Since its all shared on the M series macs, just get something that fits entirely in memory there.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
4 acronyms in this thread; the most compressed thread commented on today has 19 acronyms.
[Thread #135 for this comm, first seen 17th Oct 2026, 13:50] [FAQ] [Full list] [Contact] [Source code]
Qwen 3.8 flash next on a 4090, using strata instead of llama.cpp, gives me about 45-55 tokens/sec. Mainly used for coding tasks. The model size is perfect. If the qwen 4.0 models can be a bit less wordy in reasoning but the same size and quality they’ll be really strong.
I have been running a very simple setup but it is more impressive than I ever expected a local model to be.
My PC:
Refurbished 6900XT (16gb vram)
"Cheap" 32GB DDR5 ram
Linux Fedora
Unsloth Desktop. You can connect over cloudfare if you want, or local network, etc
Qwen 3.8 27b GSQ-RCO IQ3_XXS with 65k context @ q8, mmproj not on the GPU, mtp and fa on. I get ~50 t/s max. This is about the maximum this old GPU can handle without unplugging the monitor or slowing down the model too much.
tbh, I've been messing around with it mostly. I have been interested in its coding ability, so I have had it create some code at a few different levels of complexity. Most of these tests were nothing crazy, like a simple harness to connect to unsloth localhost models, and some HTML design tests. I have it building a UFO exploring the solar system type game, it'll be done like a week after qwen4 27b comes out!
@thebigsad1 Curious how long the first token takes once a harness sends its full system prompt into that 75k context. For agent loops, that initial wait can matter more than the 50 t/s number. Did the UFO build stay coherent across a long session, or did you have to restart it?
Played around with it on my desktop machine. Someone's distilled variant of Qwen 3.8-27B-Q4_K_M, on a intel12700k + RTX3090 24GB is what I use.
I haven't adjusted the reasoning down from very high but I mostly use it for trying tweaks to half-finished coding projects I've lost motivation to complete many years ago. I used it to develop a very simple JS-free personal website. I have secure shell which I can access from Termux or PuTTY, but sometimes I have the llama.cpp webui insecurely accessible online for one day at a time just for fun.
My server's old GPU (8gb) can't run anything useful LLW wise I don't think.
Have run some stuff locally on my main computer, but nothing setup as a proper service.
Not for complex tasks, really. Gemma4b or similar might be just enough as the backend for a voice assistant replacing Alexa (which I honestly hope, I need to get family members off of that fucking spy devices…). Small web searches might just fit. Or image recognition for Nextcloud Memories, stuff like that.
Immich's image recognition works fine off the CPU. I did get it running an LMM, but it was useless.
Epyc 7551? 256gb ddr4, r9700 + mi50. Running two models depending on what I’m doing . Llama.cpp + comfyUI or two llama.cpp for model and advisor in OMP. Main model is Qwen 3.8 27b abliterated
32GB RAM, RTX 2080 8GB VRAM, I use Tiel Coder 35b-a3b (it's a fine tune of Qwen 3.6) with Zoo Code
Your account is not tagged as a bot