Totally new to running local LLM. How are you connecting to your model via mobile?
I’m connecting to llama.cpp on my laptop through my phone via Tailscale but when my laptop sleeps I can’t access it anymore on my phone.
What are yall using for this? Thanks!
When your laptop is sleeping, it can't be doing computation. I mean, that's not specific to LLMs. You're going to have to set it in your OS to not sleep if you want to remotely access it, or to sleep under conditions that don't arise for you (e.g. "don't sleep if the lid is closed while the laptop is plugged in" or whatever).
The only exception I can think of would be if you (a) have it connected to wired Ethernet and (b) the laptop can do Wake-on-LAN; in that scenario, you could rig something up where it wakes up when something tries to contact it. In practice, you probably just don't want to have it sleeping.
Thanks! Have you ever set up wake on LAN? Any recommendations?
Nope, sorry. Also, based on the WP article, apparently there's now a wireless variant of Wake-on-LAN that I'd never heard of until now (again, assuming that the wireless chipset and the system supports it).
Thanks! I’ll do some reading.
I did, approximately forever ago. I recall having to change something in the bios and some minor issues generating the "magic packet" but overall it was less hassle than I expected. This was for a desktop.
Thanks! I’ll do some research.
Disable sleep/hibernate on your laptop. Disable sleep/hibernate/poweroff when lid is closed. If you're going to treat your laptop as a server, probably better to configure it as a server.
If I configure it as a server is there a mode that pulls very little power like sleep, but it’s ready to be activated?
Probably not. It needs to be online and the service online to be able to listen on those ports.
Got it. Thanks!
I use my homeserver for it. It's located in the broom closet and on 24/7. But there's ways to do it with a laptop. You can inhibit standby and let it run contunuously. Or configure Wake on Lan and wake it up before you use it... I mean a switched-off computer obviously can't do any computation.
Thanks! Ik interested in Wake on LAN. I don’t wanna keep drawing power when I’m not using it. Any recommendations for setting that up?
Uh, it is a bit more involved. The Arch Wiki has a lengthy article on it: https://wiki.archlinux.org/title/Wake-on-LAN
Basically, you'd enable it in your BIOS/firmware. Make sure it's enabled in the network driver. And then you need to figure out a way to send such a magic packet. You can make your router or another device in the network send it. Or do a port-forward or send it through a VPN.
Ah yeah I’m not trying to do all that. Is there a way to keep your computer almost asleep just waiting for a signal?
Laptops are designed to be fairly power-efficient. I don't know what yours does. But mine goes down to only a very few watts if idle and the display is switched off. There's the Linux tool "powertop" which shows power consumption and it can also tune most components to go to low power. Sometimes there's also a power profile setting. That shouldn't be on "performance" or anything like that. I don't think Linux has more to offer, except sleep with wake-on-lan.
Thanks! I’ll do some experimentation.
Use https://anythingllm.com/ on your mobile with its own local models? You'll be limited with your phone's hardware but it will do on a pinch where you can't reach or use your laptop. It's open source and everything works locally.
Thanks! I’ll try it out. I’m on an old phone and resistant to switch to a bigger one.