AI bubble update: 79% of companies say their GPUs sit idle at least half the time, yet most plan new AI platforms
cross-posted from: https://piefed.world/c/tech/p/1490212/ai-bubble-update-79-of-companies-say-their-gpus-sit-idle-at-least-half-the-time-yet-most
35 replies
I’d be more interested in efficiency under load.
There is evidence that some training runs at bigger companies are hilariously inefficient, or even that they run busywork to meet contract quotas. And that’s not to speak of the questionable efficiency of US “frontier models.”
And I’ve seen some staggering inefficiency even just deploying stuff, like “running Ollama on an H100 box” levels of ignorance. It boggles my mind. The heck are they even paying their employees for?
liability sponge
It's definitely a good business practice to need more startup investments just to keep the lights on. Totally normal. Literally speed running capitalism. How fast can you realize resources aren't infinite but at a global scale.
Most workers only spend ~1/3 of their time working. If you're running GPU heavy automated workflows, you're usually running batches before shifts start in the morning.
So 50% utilization seems about right, probably higher than I expect. The real question is what is their peak utilization rate, because being capped when you need it most is more important than running the things 24/7/365.
Even if you sell GPU compute, you're still pricing it amortized based on average utilization, so this isn't a deal breaker depending on the business model.
Meanwhile, open models have gotten quite good, and if you have an enterprise grade GPU you can run a lot of the good ones. I run Qwen 3.8 125b on my home server and it's acceptable. The value of owning your own GPU is getting stronger in my opinion. I guess I'll keep an eye out on used server parts or something.
As much as I think this is a bubble, fueled by effectively extortion from Nvidia and the memory makers, but initiated by Scam Altman, this isn't the needle. The needle is the data centre funding.
Also, on open models: I can run Deepseek V4 Flash at 20 tokens/s on a single 3090 desktop, a Ryzen 7800, and 128GB RAM, with measurably low quantization loss. And 4.1 once the engine updates to support it.
That was eye-opening to me. Thats faster than some cheap APIs, and higher quality too. Once companies figure that out, what are they gonna do with all those idle datacenters?
Dev time is spent on mostly-idle GPU instances too. Code, test, doesn’t work, edit, test launch again. As one sometimes needs the full environment to even test whatever they’re ultimately trying to deploy.
And yeah… like I said above, I’m more afraid of actual utilization under load. Some deployments are so inefficient it makes me want to pass out.
That's like daily driving a giant SUV because you drive eight hours to grandma's house every Christmas
90% of F350 buyers had to move a sofa once in the last 5 years.
Ironically, most of that sort of thing is done better with a minivan.
Have you never had a minivan and a pickup? I moved from Chicago to NW Florida in a minivan. I didn't think the tranny would make it, it was hangin' low for all the weight. I used to put the top down and haul lumber in my Eclipse, but a pickup is way easier.
So, just out of interest.
How many times did you move from Chicago to NW Florida?
Yes, I do in fact have both a minivan and a pickup right now. Aside from not having four wheel drive or a tow hitch, the minivan is more capable, both in terms of weight and volume (the van can haul full sheets of plywood; the pickup truck can't). And by the way, if I installed a hitch on it the minivan would be able to tow more than the truck, too.
The truck is better for hauling dirty stuff off-road, but that's about it.
Granted, my truck isn't exactly a 1-ton long-bed... but then again, neither is most people's.
I worked Lowe's outdoor lawn and garden. Many of those monster pavement princesses are hauling massive trailers. My MAGA neighbor is an asshole with a princess. But he has a yard full of trailers, car haulers and a boat you won't see him hauling everyday.
Getting a giant pickup because you might buy a washer and dryer every few years
And literally every new washer dryer comes with free delivery and install.
Bit off topic, but I can't believe people buy new appliances. For $200 each we got the nicest washer and fridge we've ever had off FB Marketplace. Our dishwasher was $50 and I found our glass-top oven for free on the road. Nothing wrong with any of those appliances.
That's just gambling with extra steps. You won on your bets, but even refurbished appliances often have expensive issues that aren't obvious until after they've been in use for a little while.
And then planning on buying a second SUV, with no second driver.
"Well, you know, the other one was getting to be about four or five years old. It was about that time."
It's so over. They leverage debt to buy these things and NVIDIA is egging them on. The datacenter companies don't have the market and even worse, they can't even install the new hardware that they are buying and GPUs are just sitting idle. They buy this on debt well on advance because every CEO is fearing that they will miss out in the future if they don't buy today. That money needs they need to paid back, but where is the money going to come from if they are not using the product? Next is Nvidia going to come in the next few years with a new product and say that all the continuously depreciating silicon that they have in their datacenters running half capacity and in warehouses sitting idle is now shit and here is the latest model, now pay up piggy. What then?
Hopefully it involves outdated datacenter GPUs flooding the open market to try and recoup what they can, selling for pennies on the dollar to anyone who will buy them, and thus people figuring out how to adapt these cards to work in consumer systems.
These cards are so specialized only the absolute enthusiasts would bother with them. They’re big, awkward, and very power hungry. Not suitable for any halfway sane person to just have sitting idle 80% of the time. The VAST majority of them will end up as ewaste.
On one hand, we're gonna need a fuckton of Chinese remanufacturing workshops putting these server chips onto regular PC boards, but on the other hand they're all export-restricted so the Chinese are probably going to be keeping any that they manage to get their hands on. So either we're going to have to start a similar remanufacturing cottage industry in the West or we're fucked.
They really aren't GPUs in the traditional sense. You can't plug them into computers and play games or do rendering. They are meant to sit in datacenter racks and do AI inference, so they are E-waste for any other application. Since we don't have enough demand for inference they will end up on dumps.
I really wonder why they’re being called GPUs and not NPUs — they aren’t primarily designed to process graphics (although they can).
The GPUs they're talking about don't have video outputs, and I think many of them aren't even PCIe. Even at pennies on the dollar, they'll still be hundreds of dollars, so way easier just to get a consumer card.
All the past infrastructure bubbles, the companies that inevitably failed at least left some usable infrastructure - rails, pipelines, dark fiber... These guys are going to crash, never pay back their loans, and the acres of datacenter, containers of GPUs, and racks of servers are just going to get recycled. It's such shocking waste that it's hard to comprehend.
Hah! I had dark fiber in mind before I hit that part of your comment. Not much of value will be left after the AI implosion.
An NVidia GB300 fills a whole rack on a server, doesn't have display outputs, draws 1.4 kW of electrical power, and requires mandatory liquid cooling capable of dissipating 1.4 kW of heat. Not a good match for a standard desktop system. I understand that they're also optimised for churning through huge amounts of low-precision floating point, aka. an AI workload, rather than being truly general-purpose cards.
If you have any use for eye-watering amounts of CUDA calculations, then they might come in handy. I remember back in the day, if you had two graphics cards in SLI, that the Batman: Arkham Asylum game gave you the option of either using both cards for rendering, or one card for rendering and one card dedicated to PhysX calculations. You could use one of these GPUs for amazingly good-looking fog effects, assuming you had another GPU capable of rendering the graphics and outputting it? Not very sensible, tho.
I suspect that unless you work in a very compute-heavy industry (AI research, but maybe aerospace?) then you won't have a use for these cards that would justify how expensive they are to run.
Don’t worry friend, our representatives in DC will ensure the money to pay for all this comes from the taxpayers. They wouldn’t have it any other way.
Data centers are simply the wrong way to approach this. The only reason for it is for billionaires to attempt full value capture of the emerging tech. The cost to run them, the cost to society and our environment - simply not worth it. I’m so glad that free open weight models are eating their lunch, and am eagerly looking forward to the AI bust cycle when we can score repurposed versions of these graphics cards for cheap to put in normal PCs again.
Fucking capitalism man, what a mistake.
Mostly with ya, but they're not running consumer graphics cards. You and I won't be able to use this tech for the power and cooling specs, let alone PC compatibility.
People say that, but given enough supply adapters will emerge. Far from impossible, they make laptops from Blackwells right now.
That reads like babbling stupid. I have to wonder if it was written by ai.
Oh, they don't work at night? Me neither. But during work time I need more compute anyway.
It's a working day somewhere