It isn't about content generation at all. It's about pattern recognition and prediction, which, in the hands of those with the most power to change the world, offers insights into our collective behavior that rulers from every age would have committed genocide to get. AI will tell them how to better build the prison the poor are being impoverished into.
"Generative" is a misnomer. It will never generate anything new, it can only regurgitate existing ideas based on patterns that already exist. It's very good at pattern recognition and summarizing, but lacks the ability to form a distinct new idea.
We are repeating an old pattern in computing: throw more hardware at the problem until efficiency becomes impossible to ignore. Bigger models have delivered remarkable gains, but they’re increasingly expensive. The next breakthroughs may come less from adding parameters and more from smarter architectures, better algorithms and more efficient inference.
DeepSeek has really led the way here, especially as they are a bit more hardware constrained. Plus they openly publish their findings and release open source models, so high hopes there.
It's probably China's play to pop the AI bubble, but I'm all for it (:
Everyone was desperate to be first because capitalism.
But we are getting good models without the insane build out requirement. Which will be hilarious to leave the cunts holding the bag. Not that the planet is better for it in the end.
~1 token per second (storage bound gen4 nvme)... Some of us have places to be.
Don't get me wrong. Its impressive that it can run at all, but honestly the usecase is exceedingly narrow. You'd have better results with a structured quantized gpu-only gemma or qwen workflow. Quality over quantity, rely on validation and a structured process: lots of cross-model review and iteration loops with spec and test driven dev. You could probably get a working alpha by the time colibri set up the environment.
Yeah I'm just beginning my local AI journey on a 5080, tried Qwen3.6 27b Q4 and was getting like 1tps because of the vram overflow. Ran it over night at it was still chewing on generating a prompt for a sub agent when I got up in the middle of the night until it simply ended in some kind of "fetch failure" lol. I think I gave it something too large to tackle, but either way 1tps is kinda garbage.
That's what I generally use. I wanted to see if I could use the 27b to "review" what the 35b put out. The 35b has been working pretty well, but it's not very thorough. I asked it to make a program and then 27b was like "this is a skeleton, there are folders but no contents." Lol
The author seems to be confusing user scalability with performance scaling:
The problem with generative AI, in the industry’s own jargon, is that it does not scale. The cost of growing from, say, a thousand users to a million is a key factor that venture capitalists examine when they evaluate start-ups.
This is a question of whether openai can handle 1 million users asking chatgpt to write a basic html website. That can be scaled horizontally and is just a matter of building more data centers.
The author then goes on to conflate this user scaling with performance scaling:
Yet the returns are diminishing. The bigger an AI model is, the less it improves with each added parameter, and so it must be made bigger at a faster rate just to sustain steady progress. I asked a few AI researchers whether they could name any other real-world software that scales so poorly. None of them could think of any. Even outside the world of software, it’s hard to find a comparable example, given that economy of scale is the principle that has made light bulbs, cars, and clothing so affordable. By economic and engineering measures, generative AI might be the worst technology ever deployed.
This is a question of whether chatgpt can generate a full complex web app. For this there may be a limit to this bigger model approach but this is common to most technologies, performance sometimes has hard limits. You aren't going to get a car to go 300 mph by making the engine bigger and adding more cylinders, there's diminishing returns, that doesn't make cars the worst technology ever deployed... maybe they are but for other reasons.
Economies of scale also isn't about performance scaling, it's about capacity scaling. Capacity scaling for AI does reflect economies of scale, that's why you have these large AI companies building large data centers.
Ok, remove the just then, the point still stands that it is a solvable problem. We know how to make data centers, it may not be easy or cheap but it's possible just like we know how to build car factories.
Yeah and the point is that model improvements so far have meant making huge increases in size, which offsets the datacenters scale out.
The whole point is that this is futile because we will always be playing catch-up to model sizes, to our ultimate downfall. The tech needs to be smarter not larger. That’s why the whole cloud AI business is shit and not going to work. as anyone with a brain has been saying from the beginning.
Jesus Christ man, people’s homes are being sized with eminent domain for this shit. It ain’t worth it.
This is only true if everyone is always using the top line model, which most people don't. Both because most people just use the default, which is a low or mid tier model, and because it's expensive. The top line models are becoming increasingly niche.
The tech needs to be smarter not larger.
I agree, that's why more focus is being put on the harness and agent orchestration these days. You can achieve better results by having a large model orchestrate a bunch of smaller model agents to do simpler tasks then trying to have the large model one shot it. This doesn't mean the whole cloud AI business is bullshit, they're still going to need to build out a lot of capacity for these smaller models and still going to need large models to handle the planning and orchestration, it just means the call count for these larger models are going to be lower.
So it's probably not going to be 1 million calls to a small model turns into 1 million calls to a larger model and the capacity never catches up, it's going to be 1 million calls to a small model and 1,000 to a large one which is more feasible to build out.
people’s homes are being sized with eminent domain for this shit. It ain’t worth it.
I don't agree with how the data centers are being rolled out, they can and should be built out with renewable energy and consent from the community which isn't happening. I disagree that data centers shouldn't be built at all or that it will be an unachievable Sisyphusian task to build them out.
I wouldn’t separate performance scalability and user scalability as they ultimately go hand in hand together.
Ok think of them as different scaling factors then, maybe n for number of requests and s for size of requests and c for complexity of requests. Scaling for n can be done horizontally by building more data centers which is possible. Scaling for s or c requires building bigger models which has diminishing returns.
Scaling for n is required to make the software business model work, like the article says. Scaling for s or c though isn't required as long as your average user keeps those constant, which is possible.
LLMs are inefficient by design.
They are less efficient when compared to what traditional computing can already do, eg. Arithmetic, structured data analysis etc. There are things that traditional computing can't do, eg. writing an essay, that can only be compared to the human brain which is hard to do. So you can say AI is inefficient at calculating 2 + 2 , but it's a hard case to make that it's inefficient at writing an essay.
Efficiency is relative, if there is a solution that uses less resources then the other solution is more efficient, but if there is no other solution then the solution is the most efficient.
Is using fable 5 to do 2+2 efficient? No, because a calculator can do that with less resources
Is using fable 5 to rewrite a code base from zig to rust efficient? Maybe since the only other solution is a human it depends on how you compare human resources to compute resources. Time wise it'll probably take the human longer since they require breaks.
Time wise it'll probably take the human longer since they require breaks.
This is only true if your only metric for 'success' is lines of code written.
LLMs will output code much faster than a human can type, there's no doubt about that. But when you consider how terrible even the frontier models still are at the architecture and maintenance side of the process, humans are still way, way more efficient.
There's a lot of variation in quality with humans though.
I don't doubt that there are some engineers better then the frontier models at coding considering architecture, maintenance performance etc. Those engineers tend to be more expensive though. I
don't think an average engineer is better then the frontier models though, and say an entry level engineer fresh out of a boot camp would be significantly worse then even a tier 2 model.
Having worked closely with junior engineers and with high-end models, I strongly disagree. In almost all cases the output of the juniors is on par or better (albeit slower) than the LLMs ,and unlike an LLM, a junior actually learns and becomes much more consistent than the AI over time.
Are you talking about a one shot from the model or using a harness? I agree a junior dev can do better then a one shot, but with a proper harness with adversarial review cycles I don't think a junior dev could
junior actually learns and becomes much more consistent than the AI over time.
A proper harness will have memory and will get more consistent the more you use it. You can "teach" it by adding skills and having it write it's learnings either locally or to repo context files.
A proper harness will have memory and will get more consistent the more you use it.
Yeah, by creating a bunch of .md files of questionable quality that get fed into an already-limited context window, on top of a pile of all sorts of other context cruft from the harness, the model provider, and whatever else is propping up the system to give the illusion of intelligence...
I've experimented with harnesses quite a bit and I don't understand the hype. The difference in quality has been middling in my experience, but the token burn has been significantly more — which makes sense when you understand that the more of the context window you use, the worse most models perform.
39 replies
Look, if you know a way to convert a PDF to text with less than 500GB of VRAM and 2000W of power used for twenty seconds, I'm all ears.
It isn't about content generation at all. It's about pattern recognition and prediction, which, in the hands of those with the most power to change the world, offers insights into our collective behavior that rulers from every age would have committed genocide to get. AI will tell them how to better build the prison the poor are being impoverished into.
"Generative" "AI" is about generation yes
"Generative" is a misnomer. It will never generate anything new, it can only regurgitate existing ideas based on patterns that already exist. It's very good at pattern recognition and summarizing, but lacks the ability to form a distinct new idea.
Paywall bypass
We are repeating an old pattern in computing: throw more hardware at the problem until efficiency becomes impossible to ignore. Bigger models have delivered remarkable gains, but they’re increasingly expensive. The next breakthroughs may come less from adding parameters and more from smarter architectures, better algorithms and more efficient inference.
DeepSeek has really led the way here, especially as they are a bit more hardware constrained. Plus they openly publish their findings and release open source models, so high hopes there.
It's probably China's play to pop the AI bubble, but I'm all for it (:
I feel like literally everyone knows this now but theres so much invested in it theres no backing out.
Too many billionaires have a need to invest and a need for future gains. It's a mental compulsion.
But it’s so good at programming if you already know how to program! Surely that’s worth burning the planet and crashing the world economy??
Actually still no
https://github.com/JustVugg/colibri
Everyone was desperate to be first because capitalism. But we are getting good models without the insane build out requirement. Which will be hilarious to leave the cunts holding the bag. Not that the planet is better for it in the end.
~1 token per second (storage bound gen4 nvme)... Some of us have places to be.
Don't get me wrong. Its impressive that it can run at all, but honestly the usecase is exceedingly narrow. You'd have better results with a structured quantized gpu-only gemma or qwen workflow. Quality over quantity, rely on validation and a structured process: lots of cross-model review and iteration loops with spec and test driven dev. You could probably get a working alpha by the time colibri set up the environment.
Yeah I'm just beginning my local AI journey on a 5080, tried Qwen3.6 27b Q4 and was getting like 1tps because of the vram overflow. Ran it over night at it was still chewing on generating a prompt for a sub agent when I got up in the middle of the night until it simply ended in some kind of "fetch failure" lol. I think I gave it something too large to tackle, but either way 1tps is kinda garbage.
You could use the 35B MoE model, tune it a little bit and get much better results. I have a 5060 ti and 70-80 tok/s are the norm
That's what I generally use. I wanted to see if I could use the 27b to "review" what the 35b put out. The 35b has been working pretty well, but it's not very thorough. I asked it to make a program and then 27b was like "this is a skeleton, there are folders but no contents." Lol
Wow I’m starting to feel bad about that time I asked AI to make a joke about scatology & eschatology sounding similar.
Is that quantized? 4 bit Qwen 3.6 can get 22tps on a 1060.
It's the q4 quantization, but it requires 20+GB vram and my 5080 only has 16
All for software that'll be out of date and fashion next year!
The author seems to be confusing user scalability with performance scaling:
This is a question of whether openai can handle 1 million users asking chatgpt to write a basic html website. That can be scaled horizontally and is just a matter of building more data centers.
The author then goes on to conflate this user scaling with performance scaling:
This is a question of whether chatgpt can generate a full complex web app. For this there may be a limit to this bigger model approach but this is common to most technologies, performance sometimes has hard limits. You aren't going to get a car to go 300 mph by making the engine bigger and adding more cylinders, there's diminishing returns, that doesn't make cars the worst technology ever deployed... maybe they are but for other reasons.
Economies of scale also isn't about performance scaling, it's about capacity scaling. Capacity scaling for AI does reflect economies of scale, that's why you have these large AI companies building large data centers.
At one of my old jobs "just" was considered a bad word
One does not simply walk into Mordor.
Ok, remove the just then, the point still stands that it is a solvable problem. We know how to make data centers, it may not be easy or cheap but it's possible just like we know how to build car factories.
Yeah and the point is that model improvements so far have meant making huge increases in size, which offsets the datacenters scale out.
The whole point is that this is futile because we will always be playing catch-up to model sizes, to our ultimate downfall. The tech needs to be smarter not larger. That’s why the whole cloud AI business is shit and not going to work. as anyone with a brain has been saying from the beginning.
Jesus Christ man, people’s homes are being sized with eminent domain for this shit. It ain’t worth it.
This is only true if everyone is always using the top line model, which most people don't. Both because most people just use the default, which is a low or mid tier model, and because it's expensive. The top line models are becoming increasingly niche.
I agree, that's why more focus is being put on the harness and agent orchestration these days. You can achieve better results by having a large model orchestrate a bunch of smaller model agents to do simpler tasks then trying to have the large model one shot it. This doesn't mean the whole cloud AI business is bullshit, they're still going to need to build out a lot of capacity for these smaller models and still going to need large models to handle the planning and orchestration, it just means the call count for these larger models are going to be lower.
So it's probably not going to be 1 million calls to a small model turns into 1 million calls to a larger model and the capacity never catches up, it's going to be 1 million calls to a small model and 1,000 to a large one which is more feasible to build out.
I don't agree with how the data centers are being rolled out, they can and should be built out with renewable energy and consent from the community which isn't happening. I disagree that data centers shouldn't be built at all or that it will be an unachievable Sisyphusian task to build them out.
The performance per parameter has been improving steadily though. Gemma 4 is ~4o level at a fraction of the parameters.
Bubble sort is also a good algorithm if we “solve” its inefficiency by using more powerful hardware. It may not be easy or cheap, but…
I wouldn’t separate performance scalability and user scalability as they ultimately go hand in hand together. LLMs are inefficient by design.
Ok think of them as different scaling factors then, maybe n for number of requests and s for size of requests and c for complexity of requests. Scaling for n can be done horizontally by building more data centers which is possible. Scaling for s or c requires building bigger models which has diminishing returns.
Scaling for n is required to make the software business model work, like the article says. Scaling for s or c though isn't required as long as your average user keeps those constant, which is possible.
They are less efficient when compared to what traditional computing can already do, eg. Arithmetic, structured data analysis etc. There are things that traditional computing can't do, eg. writing an essay, that can only be compared to the human brain which is hard to do. So you can say AI is inefficient at calculating 2 + 2 , but it's a hard case to make that it's inefficient at writing an essay.
ehhh I dunno... it's an undeniable success for a few.. and I'd wager that was the intent.
Some folks were dreaming of the Dark Enlightenment for some time, and along came the McGuffin they could use for this.
Efficiency is relative, if there is a solution that uses less resources then the other solution is more efficient, but if there is no other solution then the solution is the most efficient.
Is using fable 5 to do 2+2 efficient? No, because a calculator can do that with less resources
Is using fable 5 to rewrite a code base from zig to rust efficient? Maybe since the only other solution is a human it depends on how you compare human resources to compute resources. Time wise it'll probably take the human longer since they require breaks.
Currently it is way more cost efficient to use a human
Yea why train a human to be competent when we can rely on AI.
I'm sure in present day America that couldn't possibly go wrong lol.
This is only true if your only metric for 'success' is lines of code written.
LLMs will output code much faster than a human can type, there's no doubt about that. But when you consider how terrible even the frontier models still are at the architecture and maintenance side of the process, humans are still way, way more efficient.
There's a lot of variation in quality with humans though.
I don't doubt that there are some engineers better then the frontier models at coding considering architecture, maintenance performance etc. Those engineers tend to be more expensive though. I
don't think an average engineer is better then the frontier models though, and say an entry level engineer fresh out of a boot camp would be significantly worse then even a tier 2 model.
Having worked closely with junior engineers and with high-end models, I strongly disagree. In almost all cases the output of the juniors is on par or better (albeit slower) than the LLMs ,and unlike an LLM, a junior actually learns and becomes much more consistent than the AI over time.
Are you talking about a one shot from the model or using a harness? I agree a junior dev can do better then a one shot, but with a proper harness with adversarial review cycles I don't think a junior dev could
A proper harness will have memory and will get more consistent the more you use it. You can "teach" it by adding skills and having it write it's learnings either locally or to repo context files.
Yeah, by creating a bunch of .md files of questionable quality that get fed into an already-limited context window, on top of a pile of all sorts of other context cruft from the harness, the model provider, and whatever else is propping up the system to give the illusion of intelligence...
I've experimented with harnesses quite a bit and I don't understand the hype. The difference in quality has been middling in my experience, but the token burn has been significantly more — which makes sense when you understand that the more of the context window you use, the worse most models perform.