It just means the internet is built on a very flimsy stack of technologies and any of them failing causes huge downstream issues. We saw that with AWS, and now with Cloudflare.
It's only concerning if there are no alternatives, but as it stands there are other companies that all of these websites could have done a failover to when both AWS or Cloudflare went down. But they decided that their websites having a single point of failure was worth the risk over paying for having a proper backup system ready to go.
I like to think there was a specific person in Nebraska the author had in mind. The University there had a tap into the ARPANET back in the day and always had interesting projects going in that one wouldn't typically expect in Nebraska.
So many people seem to have just forgotten the crowd strike outage, which halted air traffic for a day and stopped a not-insignificant amount of public infrastructure
for a solid while i had forgotten cloudflare and crowdstrike were different entities, so i spent like 5 minutes scrolling through lemmy, incredibly confused
I remember experts saying 5 or 10 years ago that the increased standardization and centralization of the internet would lead to more frequent and widespread internet blackouts.
First AWS, and now this. It looks like they're right.
Two things happen when we centralize. Doesn’t matter if it’s big business or infrastructure.
Profits go up for the controlling few
consumers get fucked.
We get fucked when things go wrong, the system fails, our data gets hacked, our power goes out, our rents go up, insurance rates go up… etc etc. MegaCorps all say sorry, give us 50¢ off our next purchase and a free credit check, and carry on while we eat the losses and increasing costs.
Yes, the increasing centralization of the internet is concerning, and the fact that companies have been vibecoding stuff increases the chances of stuff going wrong. And quality control and testing aren't a priority anymore, it's as if they're just chasing short term profits. Oh wait, they are.
Imma switch my services to Bunny CDN to decrease my reliance on a huge service. And its Slovenian, so that's pretty nice.
"30 minutes"- me when I lie on the internet. Where did you get that number? You realize we can check the news and see that big sites like x and chatgpt were down like 4 hours? Not only that, they said themselves it was not an attack but a misconfiguration. News were reporting it fixed around evening utc while the issue popped up around noon. That's not a 30 min outage and is a huge failure.
The fact that Cloudflare controls half the web is concerning both for unintentional crashes like this, and for something even more insidious; what if they're coerced to cause an intentional outage should cyber war ever break out? An intentional outage for half the web in a cyber war would be devastating to put it nicely.
It's not lazy, they just spend that time on other features.
The Internet is way more secure today, partially because of centralised security efforts. Like if a site is behind Cloudflare you might aswell just not try.
Well the average website isn't going to be able to protect itself from DDoS attacks or easily provide local cache copies of its content in multiple regions all over the world or create secured tunnels protected from general attacks.
My company was affected by this and we are putting in contingency plans for this happening again but the whiteboard that we've created with all the features we need to reinvent is very full...
The snark of the following comment is not directed towards you, OP, but at the tech industry at large.
What I don't understand is why people are still surprised when this shit happens. Today, cloudflare takes down half the internet, last month it was AWS. Crowdstrike did it last year even more severely. Akamai has also caused major issues like this before, as has Google. M365/azure outages barely get reported on because they are so frequent. Yet, they are all still being used to hold up most of our infrastructure. Every single company I've done IT for has used at least one of these companies for critical infrastructure. There just aren't any other realistic options due to the refusal of non IT people to learn about IT.
If you try to use something other than one of the big companies, you're hit with one or more roadblocks.
You "don't have the budget" to selfhost. Bean counters would rather pay $100 a month indefinitely than $5k to buy new hardware that will save $1000 a month for years.
No approval for non giant corpo option, because using AWS is cheaper and has brand recognition. This is due to the same economics and myopia that caused Walmart to be one of the only places you can get groceries.
There is no other option. Every year that goes by, more small companies get gobbled up by big tech M&A. Unless your company opts to create its own implementation of a service/software, you're stuck with one of only a few options, even if you could get the approval to use something not run on big tech.
Even if you manage to jump all of the previous hurdles, the Internet connected software you're using probably relies on big tech infrastructure too. Every company has to navigate all of these hurdles for every saas/infrastructure implementation, and the only ones that successfully do it have to have leadership that not only understands why the decisions have to be made, but also need to be willing to accept the extra cost. Anyone that has dealt with upper management knows that this is exceptionally rare.
So what we are left with is a system that every professional knows is deeply broken and monopolized. The people that actually make the final decisions are largely ignorant and unwilling to invest money in fixing it, instead choosing short term savings and lack of commitment over long term security and continuity.
I mean, companies avoiding self hosting isn't just about being cheap. Cloudflare/AWS might cost $100 per mo and only have 95% uptime but you know what you're getting. Self hosting inherently introduces risk.
That 5k machine might pay for itself in half a year OR it might self destruct in 3 months. The man hours and downtime needed to unfuck that mess might cost more than multiple years of flaky cloud hosting. Alternatively, a change in data retention regulation requires hardware redundency, then next month the revenue stream from that hardware drys up and you're stuck holding a $10k loss instead of canceling a $100 payment.
I hear where you are coming from, but I think your criticisms are misdirected. For the majority of businesses, using an infrastructure provider is a sensible decision that leads to greater security and stability in the long run for less money than trying to build the same thing on their own. This isn't a decision made out of stubbornness, laziness, or ignorance about IT. It's simply that it's the better option for each individual business.
But when most companies make the decision to use an infrastructure provider, outages and risks are centralized. As you pointed out, the services you rely on are likely to use a provider even if you don't use one, so this isn't a problem that a business can solve by buying a server and hiring an IT team. These massive failures aren't a sign that businesses need to make different decisions. It's a sign that the infrastructure providers must work harder and spend more money to improve their internal isolation.
When a bridge collapses because the pedestrians happen to walk in step with the resonant frequency of the bridge, we don't blame the pedestrians for walking incorrectly or for deciding to take the bridge instead of a boat. We blame the designer of the bridge for failing to account for the mundane stresses that the bridge is expected to sustain.
Being a good CDN is an expensive exercise that requires the ability to run POPs in many countries around the world.
Cloudflare captured the market by basically being simultaneously much cheaper, better distributed and ultimately better performing than the incumbents at the time (Akamai and Limelight IIRC)
The rest of the story is capitalism doing capitalist things
The service providers get 100% of their money all the same.
This causes endless amounts of laziness on their side, and quality goes to hell.
We are causing this laziness.
Unless we, their clients, hold them accountable, and make them feel the impact of their faults in their pockets, things will continue to get worse and worse.
There was the Crowdstrike failure that tangled the airports last year, and the AWS outage that took out half the Internet just a few weeks ago. It seems like some one might be probing for vulnerabilities. One day, EVERYTHING might go down, for a while.
We'll get a chance to find out what it was like to read a book instead of a screen.
I find it at least concerning for CloudFlare's change control process. Apparently some new traffic analysis config took half the web? Maybe test things a little more?
I mean... not only is it not very concerning, I barely noticed. If not for news about it here on fediverse, I might not have known. I guess I dont visit the corpo internet all that much.
This seems interesting but I struggle to see how it helps, seems more like its for file sharing. Or is the suggestion that it could form the host side of a web ecosystem, with files for websites hosted in this decentralised way?
File sharing, a chat, backup systems, are just the low hanging fruit IMO. You can make a takedown safe "web" presence, with whatever you want on it. Decentralised, accessible with your PC off. And FOSS.
You can try it out, the new "web" WIP, if you have a couple of minutes and access yo a PC (Linux works, would love to get feedback from other platforms).
Honestly about the same as I did with crowdstrike, the AWS outages. It's not a good idea, could lead to ruin, people won't diversify, Goto step one. It's easier to just have a sensible chuckle and move on at this point.
Because they are a very good CDN and provide excellent DDos protection. They then expanded to do a whole host of other things, to the point where they do pretty much everything. Basically, they have become the first name most folks think of when they want to put something on the internet. A one stop shop for your web hosting needs. Wouldn't surprise me to learn they rent servers and VPS's as well.
Been seeing it in the selfhosting communities and subreddits for a while now. "Oh I want to put this selfhosted service on the internet. I should put it behind Cloudflare!" Most of the time it's not needed in that context.
Do you think that's concerning?
Well, they did just take out "half" of the internet today so.... In general, if it seems like "everyone" is using a single service, it's probably a good idea to see if an alternative exists and will suit your needs. Which reminds me, I should probably start looking for a replacement for Tailscale. They're starting to look a bit like Cloudflare to me, in the sense that "everyone", including myself tends to recommend them as a VPN.
Because one company offers a good service many people like, and no other company is doing it, od doing it as well as cloudflare. We are also talking about a security feature where not having it and getting hacked, may be well worse than a few hours of downtime.
Cloudflare is not necessary for the internet at all. People choose to use them.
I think it's very concerning, but not exactly surprising. It makes sense that there ends up being market leaders for digital services, because they can provide economies of scale much more than traditional services. As another commentor pointed out, they're are alternatives, even just for back-up service providers, but most sites don't pay for them.
What was more personally distressing was I realised how much I rely on lemmy when my instance, and backup instance, both went down. I'm not sure where I'd go for immediate news, especially about something niche like "why is lemmy down?". I don't use other social media and I found myself checking r/redditalternatives just to see if there was some info about the shutdown. Obviously, it was useless because reddit...
People have other things to worry about. It's concerning but there is a barrage of shit going on that this barely registers. And companies will always choose what's cheap in the short term. They believe the risk of something going wrong is small enough to warrant the possibly large impact. It's like that everywhere: in the car industry, chocolate industry, clothing industry, and so on. There's always one seemingly small decision that could fuck up the entire company but isn't worth investing in in the short term.
I wish cloudfare (whatever it is and whatever it does) had more reach and went down for longer. For so long that competitors would be considered. But alas...
It just means the internet is built on a very flimsy stack of technologies and any of them failing causes huge downstream issues. We saw that with AWS, and now with Cloudflare.
It's only concerning if there are no alternatives, but as it stands there are other companies that all of these websites could have done a failover to when both AWS or Cloudflare went down. But they decided that their websites having a single point of failure was worth the risk over paying for having a proper backup system ready to go.
Relevant XKCD, as always:
XKCD//2347
(Joke stolen from another post that's since been deleted, so reproduced here.)
I like to think there was a specific person in Nebraska the author had in mind. The University there had a tap into the ARPANET back in the day and always had interesting projects going in that one wouldn't typically expect in Nebraska.
I now imagine all the websites to fail over to the same backup services, effectively ddosing them and creating a chain reaction :D
Yeah! We call those "Cascading Failures"
They're a nightmare! 😄
And azure also went down too
So many people seem to have just forgotten the crowd strike outage, which halted air traffic for a day and stopped a not-insignificant amount of public infrastructure
for a solid while i had forgotten cloudflare and crowdstrike were different entities, so i spent like 5 minutes scrolling through lemmy, incredibly confused
DNS doesn't fail over, unfortunately.
I remember experts saying 5 or 10 years ago that the increased standardization and centralization of the internet would lead to more frequent and widespread internet blackouts.
First AWS, and now this. It looks like they're right.
Two things happen when we centralize. Doesn’t matter if it’s big business or infrastructure.
Profits go up for the controlling few
consumers get fucked.
We get fucked when things go wrong, the system fails, our data gets hacked, our power goes out, our rents go up, insurance rates go up… etc etc. MegaCorps all say sorry, give us 50¢ off our next purchase and a free credit check, and carry on while we eat the losses and increasing costs.
Are we just forgetting the gigantic Crowdstrike outage a year ago by Microsoft that halted air traffic for a full day?
Isn't crowdstrike still being sued for all the damages in aerospace? Kinda crazy
And yet, I had to remind my fiancee that it ever happened last week. People have just... forgotten.
Yes, the increasing centralization of the internet is concerning, and the fact that companies have been vibecoding stuff increases the chances of stuff going wrong. And quality control and testing aren't a priority anymore, it's as if they're just chasing short term profits. Oh wait, they are.
Imma switch my services to Bunny CDN to decrease my reliance on a huge service. And its Slovenian, so that's pretty nice.
We can see it with crowdstrike some times ago, they fucking rolled out a system-breaking update, this mean that they just builded it without testing!
What’s the fear there, that they would figure out what domain names you are resolving?
I'm guessing the concern would be resolving them to the wrong address, either to censor or to serve disinformation.
maybe you could hijack sessions by redirecting and capturing authentication i don't know im not a wizard my grandson is
Try to do secure communication without that sweet domain mame... You can't!
My thoughts is that they feel the need to control everything. And we all know how that goes usually...
.
Edit - cloudflare now says it was a misconfigured config, not a DOS attack as they initially reported
Most of the reporting I have seen suggests a massive traffic spike. Do you have some more information about the config file?
https://blog.cloudflare.com/18-november-2025-outage/
Neat!
"30 minutes"- me when I lie on the internet. Where did you get that number? You realize we can check the news and see that big sites like x and chatgpt were down like 4 hours? Not only that, they said themselves it was not an attack but a misconfiguration. News were reporting it fixed around evening utc while the issue popped up around noon. That's not a 30 min outage and is a huge failure.
The fact that Cloudflare controls half the web is concerning both for unintentional crashes like this, and for something even more insidious; what if they're coerced to cause an intentional outage should cyber war ever break out? An intentional outage for half the web in a cyber war would be devastating to put it nicely.
You know back in my day websites would protect themselves, as was the style at the time.
Now a days they just get cloudflare and put up a cookie notice.
Just one of those things lazy devs do.
Why reinvent the wheel.
It's not lazy, they just spend that time on other features.
The Internet is way more secure today, partially because of centralised security efforts. Like if a site is behind Cloudflare you might aswell just not try.
Ya. Like popups and cookie notices.
Well the average website isn't going to be able to protect itself from DDoS attacks or easily provide local cache copies of its content in multiple regions all over the world or create secured tunnels protected from general attacks. My company was affected by this and we are putting in contingency plans for this happening again but the whiteboard that we've created with all the features we need to reinvent is very full...
The snark of the following comment is not directed towards you, OP, but at the tech industry at large.
What I don't understand is why people are still surprised when this shit happens. Today, cloudflare takes down half the internet, last month it was AWS. Crowdstrike did it last year even more severely. Akamai has also caused major issues like this before, as has Google. M365/azure outages barely get reported on because they are so frequent. Yet, they are all still being used to hold up most of our infrastructure. Every single company I've done IT for has used at least one of these companies for critical infrastructure. There just aren't any other realistic options due to the refusal of non IT people to learn about IT.
If you try to use something other than one of the big companies, you're hit with one or more roadblocks.
You "don't have the budget" to selfhost. Bean counters would rather pay $100 a month indefinitely than $5k to buy new hardware that will save $1000 a month for years.
No approval for non giant corpo option, because using AWS is cheaper and has brand recognition. This is due to the same economics and myopia that caused Walmart to be one of the only places you can get groceries.
There is no other option. Every year that goes by, more small companies get gobbled up by big tech M&A. Unless your company opts to create its own implementation of a service/software, you're stuck with one of only a few options, even if you could get the approval to use something not run on big tech.
Even if you manage to jump all of the previous hurdles, the Internet connected software you're using probably relies on big tech infrastructure too. Every company has to navigate all of these hurdles for every saas/infrastructure implementation, and the only ones that successfully do it have to have leadership that not only understands why the decisions have to be made, but also need to be willing to accept the extra cost. Anyone that has dealt with upper management knows that this is exceptionally rare.
So what we are left with is a system that every professional knows is deeply broken and monopolized. The people that actually make the final decisions are largely ignorant and unwilling to invest money in fixing it, instead choosing short term savings and lack of commitment over long term security and continuity.
For a lot of people who would self host, $100 at a time is easier to get together than a few thousand at once.
Not talking about individuals, homie. Talking about companies.
I mean, companies avoiding self hosting isn't just about being cheap. Cloudflare/AWS might cost $100 per mo and only have 95% uptime but you know what you're getting. Self hosting inherently introduces risk.
That 5k machine might pay for itself in half a year OR it might self destruct in 3 months. The man hours and downtime needed to unfuck that mess might cost more than multiple years of flaky cloud hosting. Alternatively, a change in data retention regulation requires hardware redundency, then next month the revenue stream from that hardware drys up and you're stuck holding a $10k loss instead of canceling a $100 payment.
I hear where you are coming from, but I think your criticisms are misdirected. For the majority of businesses, using an infrastructure provider is a sensible decision that leads to greater security and stability in the long run for less money than trying to build the same thing on their own. This isn't a decision made out of stubbornness, laziness, or ignorance about IT. It's simply that it's the better option for each individual business.
But when most companies make the decision to use an infrastructure provider, outages and risks are centralized. As you pointed out, the services you rely on are likely to use a provider even if you don't use one, so this isn't a problem that a business can solve by buying a server and hiring an IT team. These massive failures aren't a sign that businesses need to make different decisions. It's a sign that the infrastructure providers must work harder and spend more money to improve their internal isolation.
When a bridge collapses because the pedestrians happen to walk in step with the resonant frequency of the bridge, we don't blame the pedestrians for walking incorrectly or for deciding to take the bridge instead of a boat. We blame the designer of the bridge for failing to account for the mundane stresses that the bridge is expected to sustain.
It even took down the very instance we're posting this on.
You must not have been paying attention when that one Java programmer quit and took his code with him.
I didn't even notice that it went down.
It took down a fifth of the Internet, not half.
I found two websites that didn't work, that's it.
A third of the “top 100” were in that 1/5th total. Most websites I personally wanted were down, including lemmy for me.
Lemmy and ipchicken.com were down for me.
Being a good CDN is an expensive exercise that requires the ability to run POPs in many countries around the world.
Cloudflare captured the market by basically being simultaneously much cheaper, better distributed and ultimately better performing than the incumbents at the time (Akamai and Limelight IIRC)
The rest of the story is capitalism doing capitalist things
The service providers get 100% of their money all the same.
This causes endless amounts of laziness on their side, and quality goes to hell.
We are causing this laziness.
Unless we, their clients, hold them accountable, and make them feel the impact of their faults in their pockets, things will continue to get worse and worse.
There was the Crowdstrike failure that tangled the airports last year, and the AWS outage that took out half the Internet just a few weeks ago. It seems like some one might be probing for vulnerabilities. One day, EVERYTHING might go down, for a while.
We'll get a chance to find out what it was like to read a book instead of a screen.
I find it at least concerning for CloudFlare's change control process. Apparently some new traffic analysis config took half the web? Maybe test things a little more?
I mean... not only is it not very concerning, I barely noticed. If not for news about it here on fediverse, I might not have known. I guess I dont visit the corpo internet all that much.
I have been concerned recently that despite my best efforts I am still too attached to the corporately owned internet.
The fact that I felt no impact from this was a nice treat to start my week.
Most corporate IT hires fucking morons as admins :(
Where do I get a fucking moron certificate?
It would be nice to be hired atm.
Amazon i think
I think you need to possess the certificate first before they'll even consider you, though.
Also, hopefully without actually having to be a fucking moron, to get the cert.
You know? The fake sorts
More than likely it's their management who are the morons.
One does not exclude the other :/
I've been a contractor most of my career and I can assure you, fair share of ID10Ts in my circles as well.
Can confirm it's not industry specific.
Places keep hiring me.
You must be likeable then.
What's that like?
Sometimes there's a high number next to my name on social media sites.
Those are the ones that get occasionally hired as admins for someone's corporate IT ;)
So, and I'm gonna pull my shameless plug ofc, but what about a decentralised internet?
Check out tenfingers or the sub (I put the weblink, is it !lemmy.world/c/tenfingers on lemmy browser apps?).
What about we take the internet back?
This seems interesting but I struggle to see how it helps, seems more like its for file sharing. Or is the suggestion that it could form the host side of a web ecosystem, with files for websites hosted in this decentralised way?
Exactly.
File sharing, a chat, backup systems, are just the low hanging fruit IMO. You can make a takedown safe "web" presence, with whatever you want on it. Decentralised, accessible with your PC off. And FOSS.
You can try it out, the new "web" WIP, if you have a couple of minutes and access yo a PC (Linux works, would love to get feedback from other platforms).
Honestly about the same as I did with crowdstrike, the AWS outages. It's not a good idea, could lead to ruin, people won't diversify, Goto step one. It's easier to just have a sensible chuckle and move on at this point.
Because they are a very good CDN and provide excellent DDos protection. They then expanded to do a whole host of other things, to the point where they do pretty much everything. Basically, they have become the first name most folks think of when they want to put something on the internet. A one stop shop for your web hosting needs. Wouldn't surprise me to learn they rent servers and VPS's as well.
Been seeing it in the selfhosting communities and subreddits for a while now. "Oh I want to put this selfhosted service on the internet. I should put it behind Cloudflare!" Most of the time it's not needed in that context.
Well, they did just take out "half" of the internet today so.... In general, if it seems like "everyone" is using a single service, it's probably a good idea to see if an alternative exists and will suit your needs. Which reminds me, I should probably start looking for a replacement for Tailscale. They're starting to look a bit like Cloudflare to me, in the sense that "everyone", including myself tends to recommend them as a VPN.
It's good that it goes down once in a while, so that people notice.
Because one company offers a good service many people like, and no other company is doing it, od doing it as well as cloudflare. We are also talking about a security feature where not having it and getting hacked, may be well worse than a few hours of downtime.
Cloudflare is not necessary for the internet at all. People choose to use them.
I barely noticed
I didn't, just people bitching after
I am, but what can we do about it? We don't run things.
Not only does it make things brittle, but it makes a smaller number of things for governments to intercept to scan all internet traffic.
Amazon the other week, then this. Really does show how vulnerable much of the net is
it's very concerning, but what are the alternatives?
Someone please give a serious answer please.
My selfhosting gimmick is “no reliance on things outside my control”
I have to look into something like this, thanks for the resource
Well, lemmy. That worked fine.
my lemmy was down
You need to create your own Lemmy, with blackjack and hookers.
Multiple instances gives redundancy
Fastly?
I think it's very concerning, but not exactly surprising. It makes sense that there ends up being market leaders for digital services, because they can provide economies of scale much more than traditional services. As another commentor pointed out, they're are alternatives, even just for back-up service providers, but most sites don't pay for them.
What was more personally distressing was I realised how much I rely on lemmy when my instance, and backup instance, both went down. I'm not sure where I'd go for immediate news, especially about something niche like "why is lemmy down?". I don't use other social media and I found myself checking r/redditalternatives just to see if there was some info about the shutdown. Obviously, it was useless because reddit...
People have other things to worry about. It's concerning but there is a barrage of shit going on that this barely registers. And companies will always choose what's cheap in the short term. They believe the risk of something going wrong is small enough to warrant the possibly large impact. It's like that everywhere: in the car industry, chocolate industry, clothing industry, and so on. There's always one seemingly small decision that could fuck up the entire company but isn't worth investing in in the short term.
I wish cloudfare (whatever it is and whatever it does) had more reach and went down for longer. For so long that competitors would be considered. But alas...