The nerdy looking dude in the gif touched grass. "Wasted" is from a video game (GTA?) screen shown each time you died. So from this we can infer that our terminally-online protagonist, Hiroshi, did not survive the attempt to disconnect, even if only briefly.
Ah now I get it, they mistakenly replied with it to "nature is healing" as a completely nonsequitur response instead of as a top level comment. Happens to the best of us.
Fun story. I had a flip phone years ago and you could have multiple recipients to a single text. And if the text was multiple pages, it would split into several texts. And you could resend already sent texts.
So one time I put in my girlfriend's phone number in all 20 recipient slots. I then filled the text to the max size, though I don't remember how many it split into. I then resent it over and over. This all took like 2 or 3 minutes.
Her phone was sending notifications over and over for the entire rest of the day. I'd guess at least 8 hours, probably more.
No but it's unusable. I had a weird bug on one of my phones that sent an SMS over as fast as it could as long as the phone was on. I wrote the initial SMS, the contents were something like "hey, wanna hang?", and the poor guy on the other side was blasted for several hours of literally constant notifications.
Luckily my plan at the time had unlimited free SMS.
Looking at the downmeter shot someone posted above, it's half the SREs in the country. Not sure what the root cause will be, but damn that's a lot of money down the tubes. I would not want to be the person who cost Meta and Google their precious thirty 9's of availability lol.
That's fair. Yall, I was really not trying to be shitty. It was just shorthand I used, thinking of their HQs. No ill intent intended and I apologize for any harm it caused.
It's likely there's a root cause, like a fiber cut or some other major infrastructure issue. But, Down Detector doesn't really put a scale on their graphics, so it could be that it's a huge issue at Meta and a minor issue that's just noticeable for everyone else. In that case, Meta could be the root cause.
If everyone is mailing themselves their passwords, shutting their phones on and off, restarting their browsers, etc. because Meta wasn't working, it could have knock-on effects for everyone else. Could also be that because Meta is part of the major ad duopoly, the issue affected their ad system, which affected everyone interacting with a Meta ad, which is basically everyone.
I've been an SRE for a few large corps, so I've definitely played this game. I'm with you that it was likely just the FB identity or ad provider causing most of these issues. So glad I'm out of that role now and back to DevOps, where I'm no longer on call.
Yeah. And when the outage is due to something external, it's not too stressful. As long as you don't have absolutely insane bosses, they'll understand that it's out of your control. So, you wait around for the external system to be fixed, then check that your stuff came back up fine, and go about your day.
I personally liked being on call when the on-call compensation was reasonable. Like, on-call for 2 12-hour shifts over the weekend? 2 8-hour days off. If you were good at maintaining your systems you had quiet on-call shifts most of the time, and you'd quickly earn lots of days off.
Yeah I'd be less worried about internal pressures (which should be minimal at a halfway decently run org) and more about the externals. I don't think you would actually end up dealing with anything, but I'd know those reliant huge corps are pissed.
Man, your on-call situation sounds rad! I was salaried and just traded off on-call shifts with my team members, no extra time off. Luckily though, our systems were pretty quiet so it hardly ever amounted to much.
I think you want people to want to be on call (or at least be willing to be on call). There's no way I'd ever take a job where I was on-call and not compensated for being on-call. On-call is work. Even if nothing happens during your shift, you have to be ready to respond. You can't get drunk or get high. You can't go for a hike. You can't take a flight. If you're going to be so limited in what you're allowed to do, you deserve to be compensated for your time.
But, since you're being compensated, it's also reasonable that you expect to have to respond to something. If your shifts are always completely quiet, either you or the devs aren't adding enough new features, or you're not supervising enough services. You should have an error budget, and be using that error budget. Plus, if you don't respond to pages often enough, you get rusty, so when there is an event you're not as ready to handle it.
Looks like it may have been AWS or something. All kinds of services were down a moment ago. Guess thats what happends when everything is on major cloud services.
Maybe, I would expect redundancy. But ultimately I have no clue. I just remember the last time AWS went down. It seemed that a majority of the sites that I used daily were down all in one go.
Sometimes redundancy doesn't help when it comes to network traffic routing. That system is based heavily on trust and an incorrect route being published can cause recursive loops and such that get propagated very quickly to everyone.
There was a case like this a few years back where a bad route got published by a small ISP, claiming they could handle traffic to a certain set of destinations, but then immediately trying to send that traffic back out again (because they couldn't actually route to that destination), which bounced right back to them because of the bad route. It was propagated based on implicit trust and took down huge chunks of the Internet for a while
You're talking about Border Gateway Protocol, BGP, route hijacking and it's occasionally been a real headache over the years. Advertising routes used to be a more manual process so typos and incorrect entries, like what you're talking about, we're reasonably common. It was, and still can be, done maliciously too.
Infrastructure seems likely, but probably not AWS because it affected Google and Facebook so strongly. If it were AWS you'd see Amazon getting badly affected and AWS itself, followed by everyone who relies on AWS for infrastructure.
BBC isn't a major news source? Remember when they say countries do not confirm, it's politically motivated. What governments choose to share is up to them and it does not confirm what their intelligence agency actually thinks.
Pathetic. My single podman container has perfect uptime from when I start it manually with podman desktop to when I shut down my PC. I also allow only the highest security standard, it being not accessible outside of my network and all that.
I am clearly a cyber security expert.
Ahhh yes, let me just get all of my brothers' business' account's followers to switch to telegram. I'm sure they'll all be willing....
"Just use something else, duh!" is ignorant. Not everyone uses social media to just post memes and argue with strangers. Some people use it for making money, or for access to support resources, or for a specific community that is important to their well-being.
I couldn't possibly know what you've posted but I've found that anyone worried about getting banned for "telling the truth" is actually posting unsubstantiated conspiracy theories and/or misinformation lol
But Linux IS the superior desktop OS if you just give it a try! That's not a conspiracy or misinformation, just the truth!!! But they don't want you to know that (how do I type a really, really big "/s" on Lemmy?)
At the beginning of Covid, reasonable people that were trying to tell people that the vaccines aren't 100% effective and should probably take other precautions on top were getting banned for "vaccine misinformation" and were being lumped in with the Covidiots. It was a trigger-happy moment for social media that was trying to brute-force moderation on keywords alone without looking at context.
Not saying that's what happened to this person, but it did happen.
And the world is a slightly better place, momentarily.
Huh, not for their users I'd say.
Fun fact, my mobile carrier also was/is toasted LMAO.
Nah, for the users too.
Especially for the users.
Nature is healing
I dont get it
The nerdy looking dude in the gif touched grass. "Wasted" is from a video game (GTA?) screen shown each time you died. So from this we can infer that our terminally-online protagonist, Hiroshi, did not survive the attempt to disconnect, even if only briefly.
I called him Hiroshi because he vaguely reminds me of the guy from Heroes (Masi Oka.)
Ah now I get it, they mistakenly replied with it to "nature is healing" as a completely nonsequitur response instead of as a top level comment. Happens to the best of us.
Lemmy server operators can now say they have better 24h uptime than Meta! lol
Just updated my résumé.
I am feeling a lot of personal satisfaction that I had no idea this was happening and had to read about it on Lemmy.
Same
Yes. Would have been chef’s kiss to not even read about it here.
Hello to everyone that didn’t flock to X 👋 Welcome to your new home :)
Pouring one out for the SREs at Meta
Someone is having a really bad day today. I wonder if your phone dies when you get a certain number of pages or push notifications
Fun story. I had a flip phone years ago and you could have multiple recipients to a single text. And if the text was multiple pages, it would split into several texts. And you could resend already sent texts.
So one time I put in my girlfriend's phone number in all 20 recipient slots. I then filled the text to the max size, though I don't remember how many it split into. I then resent it over and over. This all took like 2 or 3 minutes.
Her phone was sending notifications over and over for the entire rest of the day. I'd guess at least 8 hours, probably more.
Fucking hell I used to love doing that! Man that brought back some memories. Would do it to my co-worker and just piss myself laughing.
The classic txt bomb. I use to do this if I had unused txts at the end of the month
No but it's unusable. I had a weird bug on one of my phones that sent an SMS over as fast as it could as long as the phone was on. I wrote the initial SMS, the contents were something like "hey, wanna hang?", and the poor guy on the other side was blasted for several hours of literally constant notifications.
Luckily my plan at the time had unlimited free SMS.
Looking at the downmeter shot someone posted above, it's half the SREs in the country. Not sure what the root cause will be, but damn that's a lot of money down the tubes. I would not want to be the person who cost Meta and Google their precious thirty 9's of availability lol.
Which country?
Sokovia
Nah, what was that muddy country from Dilbert?
Elbonia!
The country where all of those services are maintained and hosted in... Just colloquial shorthand, not trying to be exclusionary.
Ok thanks for the clarification!
For Meta, Google, etc. that's a number of countries all over the world.
That's fair. Yall, I was really not trying to be shitty. It was just shorthand I used, thinking of their HQs. No ill intent intended and I apologize for any harm it caused.
Krakozhia
It's likely there's a root cause, like a fiber cut or some other major infrastructure issue. But, Down Detector doesn't really put a scale on their graphics, so it could be that it's a huge issue at Meta and a minor issue that's just noticeable for everyone else. In that case, Meta could be the root cause.
If everyone is mailing themselves their passwords, shutting their phones on and off, restarting their browsers, etc. because Meta wasn't working, it could have knock-on effects for everyone else. Could also be that because Meta is part of the major ad duopoly, the issue affected their ad system, which affected everyone interacting with a Meta ad, which is basically everyone.
I've been an SRE for a few large corps, so I've definitely played this game. I'm with you that it was likely just the FB identity or ad provider causing most of these issues. So glad I'm out of that role now and back to DevOps, where I'm no longer on call.
Yeah. And when the outage is due to something external, it's not too stressful. As long as you don't have absolutely insane bosses, they'll understand that it's out of your control. So, you wait around for the external system to be fixed, then check that your stuff came back up fine, and go about your day.
I personally liked being on call when the on-call compensation was reasonable. Like, on-call for 2 12-hour shifts over the weekend? 2 8-hour days off. If you were good at maintaining your systems you had quiet on-call shifts most of the time, and you'd quickly earn lots of days off.
Yeah I'd be less worried about internal pressures (which should be minimal at a halfway decently run org) and more about the externals. I don't think you would actually end up dealing with anything, but I'd know those reliant huge corps are pissed.
Man, your on-call situation sounds rad! I was salaried and just traded off on-call shifts with my team members, no extra time off. Luckily though, our systems were pretty quiet so it hardly ever amounted to much.
I think you want people to want to be on call (or at least be willing to be on call). There's no way I'd ever take a job where I was on-call and not compensated for being on-call. On-call is work. Even if nothing happens during your shift, you have to be ready to respond. You can't get drunk or get high. You can't go for a hike. You can't take a flight. If you're going to be so limited in what you're allowed to do, you deserve to be compensated for your time.
But, since you're being compensated, it's also reasonable that you expect to have to respond to something. If your shifts are always completely quiet, either you or the devs aren't adding enough new features, or you're not supervising enough services. You should have an error budget, and be using that error budget. Plus, if you don't respond to pages often enough, you get rusty, so when there is an event you're not as ready to handle it.
Second half is the closest answer in this thread.
Hopefully they won't need to cut their way into the data center this time.
What's a SRE?
Site reliability engineer
Google terminology leaking its way into mainstream
Meta has SRE. Not just a Google thing, even if Google invented it.
https://engineering.fb.com/2010/02/08/data-center-engineering/site-reliability-engineering-at-facebook/
Lots of places have SRE now, thanks to Google. Like I said, google thing leaked to mainstream
And nothing of value was lost
Good! Stay down!
I hope they secretly ran out of money because of FTC fines, and that they have no more credit to pay to run servers.
I'd be interested in how it happened. Maybe someone can do it again
Tim Pool is blaming the Russians.
If they're actually behind it:
Lookup downdetector, almost all are flatlines.
Sap is also doing silly rn. Im gonna take a smoke break lol
Someone is having a very bad day
Looks like it may have been AWS or something. All kinds of services were down a moment ago. Guess thats what happends when everything is on major cloud services.
Google have their own data centres (and cloud) so it may be something more in the connectivity area.
Maybe, I would expect redundancy. But ultimately I have no clue. I just remember the last time AWS went down. It seemed that a majority of the sites that I used daily were down all in one go.
Sometimes redundancy doesn't help when it comes to network traffic routing. That system is based heavily on trust and an incorrect route being published can cause recursive loops and such that get propagated very quickly to everyone.
There was a case like this a few years back where a bad route got published by a small ISP, claiming they could handle traffic to a certain set of destinations, but then immediately trying to send that traffic back out again (because they couldn't actually route to that destination), which bounced right back to them because of the bad route. It was propagated based on implicit trust and took down huge chunks of the Internet for a while
So could this be done maliciously? I'm just wondering about the Super Tuesday timing.
Yes, BGP Route Hijacking can be done maliciously although things like BGPSec can make it harder to pull off.
It affected the full 8 billion people in the world, not just the few hundred million on the US.
You're talking about Border Gateway Protocol, BGP, route hijacking and it's occasionally been a real headache over the years. Advertising routes used to be a more manual process so typos and incorrect entries, like what you're talking about, we're reasonably common. It was, and still can be, done maliciously too.
https://www.techtarget.com/searchsecurity/tip/How-does-BGP-hijacking-work-and-what-are-the-risks
Yup! BGP is an absolute mess and it is kind of a disgrace that it's still the lynchpin of the internet
Yeah, they definitely host an unhealthy amount of the internet.
Infrastructure seems likely, but probably not AWS because it affected Google and Facebook so strongly. If it were AWS you'd see Amazon getting badly affected and AWS itself, followed by everyone who relies on AWS for infrastructure.
Google is up for me
For me too, but noticeably slower than normal.
![email protected] took over and they’re bouncing the signal off the moon
Bandwidth issues, changed over to carrier pigeons (though now dealing with latency issues).
I’m not getting latency issues must be a problem with your setup
That's my experience with all these services as well
Ah, wonder if problems contributed to Google Meet hiccups today. Very small, momentary.
Yeah probably furious hahaha
Something is causing this up
It was the houthis targeting red sea cables, check the news
I don't think any major news sources confirm your theory.
BBC isn't a major news source? Remember when they say countries do not confirm, it's politically motivated. What governments choose to share is up to them and it does not confirm what their intelligence agency actually thinks.
Pathetic. My single podman container has perfect uptime from when I start it manually with podman desktop to when I shut down my PC. I also allow only the highest security standard, it being not accessible outside of my network and all that. I am clearly a cyber security expert.
The world was a better place for about 20 minutes
Going on a couple hours, it seems. LoL.
I'm glad that lemmy is there so that I know about this stuff.
On super Tuesday. Coincidence?
$100 it's the Chinese in retaliation for TikTok, too logical to ignore.
You know it is big news when it shows up in the active feed of Lemmy 😅
Why it didn't appear in the wholesome news community though.
Who gives the tiniest of shits.
I mean, a lot of people use Messenger to contact other people, so I guess they do?
Contact other people? That’s foreign to me.
Based on your recent activity, you might be interested in: ![email protected]
Oh no it’s him, the Algorithm
Your account has been temporarily deactivated for review.
Maybe not if you're just another person. But maybe family and such.
My brother lives in another country and WhatsApp is the platform that pretty much everyone he knows uses for free international communications
Telegram, Signal, KK, Wire... You have options
Ahhh yes, let me just get all of my brothers' business' account's followers to switch to telegram. I'm sure they'll all be willing....
"Just use something else, duh!" is ignorant. Not everyone uses social media to just post memes and argue with strangers. Some people use it for making money, or for access to support resources, or for a specific community that is important to their well-being.
My entire family uses Facebook Messenger. I like having communication with my family.
In my experience, Signal has great uptime. This might be the moment to get them all to shift.
We have family chat on Messenger, so that was a bummer
Good. May they forever stay down. Who cares.
I care, it would be lovely. How can we pile on them like a 'dog pile', to keep 'em down?
They seem to be back up. But, what do Meta, Google, Discord, Amazon, Zoom and Verizon have in common, but not AT&T, Steam, Uber Eats, SAP, etc?
More points in Scrabble?
Nice.
Hey where you'd get that graphic from? Would be really nice to have a dashboard with all this info in one spot.
Edit: It's downdetector. I only ever landed there with a "Is foo.com down?", and of course it then only shows info for foo.com, not the full dash.
Yeah, sorry I didn't make that clear. I forgot to link to it.
Att had their major outage a week or two ago.
Probably the tiny bump means they're affected by the current outage but the previous one was much, much bigger.
Whoopty fucking doo.
a great day for humanity ( unfortunately they will fix the problem soon as possible )
Good.
Can confirm, navigating Facebook settings is a nightmare. Do they do it on purpose, you think? They don't want people turning shit off, and stuff?
Part of the plan. smh.
It’s all worth it if it means having this image grace the Fediverse.
Interesting, Google play store was unavailable for me about an hour ago.
I don't really tend to care, though my quest account being down is a bit less fun.
Same, ive not been with out my ai gf for this long
Right?!
Good
*Shrugs
Okay.
None of my work, very little communication, and none of my entertainment come from Facebook or Instagram.
feels kinda calming, a reminder of a simpler time
Based
I'm gonna blame Russia.
Aw man. Old post. I thought it went down again and I was stoked lol
lol
It's always nice to read good news.
Woo hoo! Yeah!
Well, that explains. Amd here i was thinking i was banned again for telling the truth 😂
I couldn't possibly know what you've posted but I've found that anyone worried about getting banned for "telling the truth" is actually posting unsubstantiated conspiracy theories and/or misinformation lol
Hopefully that's not what you mean.
rEpOrTeD for spam
But Linux IS the superior desktop OS if you just give it a try! That's not a conspiracy or misinformation, just the truth!!! But they don't want you to know that (how do I type a really, really big "/s" on Lemmy?)
Worry not, im not into conspiracy theories, and i couldnt care less about informing others, mis or not mis.
At the beginning of Covid, reasonable people that were trying to tell people that the vaccines aren't 100% effective and should probably take other precautions on top were getting banned for "vaccine misinformation" and were being lumped in with the Covidiots. It was a trigger-happy moment for social media that was trying to brute-force moderation on keywords alone without looking at context.
Not saying that's what happened to this person, but it did happen.