What's your use case? I've found Qwen 3.5 4b to be fairly useless. Decent for like summarizing stuff or generating annotations on things, but that's about it.
I don’t know, I was just kinda playing around to see if it would work at all. I think the laptop is a bit too old for anything. But I can use my desktop to build python scripts that I can run on the laptop maybe.
If the kink you're referring to is the amount of time Qwen's "thinking" about your greeting, that's not necessarily actually a problem. I've seen common misconceptions about exactly this kind of scenario.
Models like these are trained to follow instructions. When they receive a user input they think "what is the user asking me to do, and how do I go about doing it?"
If all you've said is "Hi", that leaves them with a huge puzzle and no hints how to solve it. What does the user want them to do? Is this a test? Did they start writing an actual instruction and accidentally hit enter partway through? Do they want me to write a "hello world" app, and if so in what language? Is this the outout of a "hello world" app that I wrote in a previous round that I now need to check up on?
Qwen's models are known to be rather "thinky" in the first place - it's part of how they're able to hit so high above their weight class - so it's not surprising that it's taking its time to figure out this challenge you've presented it with.
Oh thank fuck. I thought I was going crazy...
My first attempt at local models was a Qwen. A base model. I swear to god all I said was hello and the thing started and never stopped. I guess that's what a filibuster must be like? It was both amazing, yet terrifying.
I've a bit more xp now, but yeah Qwen and really really thinky, dear gods !
My colleague got himself an abliterated one and just leave it to run overnight trying to pentest shit in his home network, but I guess I prefer something a bit more hands on, myself.
Heh. I use Qwen3.8-27B as my main coding agent, and I sometimes feel really bad for it as I watch the stream of thought go by filled with "wait... am I doing this right? What if I'm completely wrong about this? Is this really what I should be doing??" I want to pet it and assure it that it should have more confidence in itself. :)
But it's the tradeoff that Qwen made to cram that much smarts into such a small bundle of parameters, I guess.
Yeah, I can see that but it's horrendously slow, I find. I tried a 3.8 online and I just dont' like the way it works, overall :(
It goes on and on and on and never stops to ask despite that being one of my main rules.
There's a fine-tuned version called Swift that's been trained to use 58.3% fewer reasoning tokens, apparently with almost no loss of quality. I've not been in a hurry with my personal usage (I set my coding agent loose on a task and then go have tea or whatever) so I haven't experimented with it much but it might be worth checking out if the verbosity of reasoning is a problem.
Qwen4 will be coming out soon too, it has a 27B version, so perhaps they'll have incorporated more efficient reasoning into it from the start. I'm looking forward to testing it out.
12 replies
What's your use case? I've found Qwen 3.5 4b to be fairly useless. Decent for like summarizing stuff or generating annotations on things, but that's about it.
I don’t know, I was just kinda playing around to see if it would work at all. I think the laptop is a bit too old for anything. But I can use my desktop to build python scripts that I can run on the laptop maybe.
Then 4b or even smaller is probably fine, it’s surprising what even sub 1b models can do if you have a good use case
If the kink you're referring to is the amount of time Qwen's "thinking" about your greeting, that's not necessarily actually a problem. I've seen common misconceptions about exactly this kind of scenario.
Models like these are trained to follow instructions. When they receive a user input they think "what is the user asking me to do, and how do I go about doing it?"
If all you've said is "Hi", that leaves them with a huge puzzle and no hints how to solve it. What does the user want them to do? Is this a test? Did they start writing an actual instruction and accidentally hit enter partway through? Do they want me to write a "hello world" app, and if so in what language? Is this the outout of a "hello world" app that I wrote in a previous round that I now need to check up on?
Qwen's models are known to be rather "thinky" in the first place - it's part of how they're able to hit so high above their weight class - so it's not surprising that it's taking its time to figure out this challenge you've presented it with.
Oh thank fuck. I thought I was going crazy... My first attempt at local models was a Qwen. A base model. I swear to god all I said was hello and the thing started and never stopped. I guess that's what a filibuster must be like? It was both amazing, yet terrifying. I've a bit more xp now, but yeah Qwen and really really thinky, dear gods ! My colleague got himself an abliterated one and just leave it to run overnight trying to pentest shit in his home network, but I guess I prefer something a bit more hands on, myself.
Heh. I use Qwen3.8-27B as my main coding agent, and I sometimes feel really bad for it as I watch the stream of thought go by filled with "wait... am I doing this right? What if I'm completely wrong about this? Is this really what I should be doing??" I want to pet it and assure it that it should have more confidence in itself. :)
But it's the tradeoff that Qwen made to cram that much smarts into such a small bundle of parameters, I guess.
Yeah, I can see that but it's horrendously slow, I find. I tried a 3.8 online and I just dont' like the way it works, overall :( It goes on and on and on and never stops to ask despite that being one of my main rules.
There's a fine-tuned version called Swift that's been trained to use 58.3% fewer reasoning tokens, apparently with almost no loss of quality. I've not been in a hurry with my personal usage (I set my coding agent loose on a task and then go have tea or whatever) so I haven't experimented with it much but it might be worth checking out if the verbosity of reasoning is a problem.
Qwen4 will be coming out soon too, it has a 27B version, so perhaps they'll have incorporated more efficient reasoning into it from the start. I'm looking forward to testing it out.
This is time to first token, which is related to pre filling the cache and weights, not so much thinking.
4b why?
I'm trying to use an old HP zBook. I was told it would be faster.
what are the specs? Try Ling 3.0 Tiny, that's probably the fastest decent model, it should be both smarter and faster than Qwen 3.5 4b