Major IT outage affecting banks, airlines, media outlets across the world
All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It's all very exciting, personally, as someone not responsible for fixing it.
Apparently caused by a bad CrowdStrike update.
Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We'll see if that changes over the weekend...
https://www.abc.net.au/news/2024-07-19/technology-shutdown-abc-media-banks-institutions/104119960Open linkView original on lemmy.ml1204
Comments541
Reading into the updates some more... I'm starting to think this might just destroy CloudStrike as a company altogether. Between the mountain of lawsuits almost certainly incoming and the total destruction of any public trust in the company, I don't see how they survive this. Just absolutely catastrophic on all fronts.
If all the computers stuck in boot loop can't be recovered... yeah, that's a lot of cost for a lot of businesses. Add to that all the immediate impact of missed flights and who knows what happening at the hospitals. Nightmare scenario if you're responsible for it.
This sort of thing is exactly why you push updates to groups in stages, not to everything all at once.
Looks like the laptops are able to be recovered with a bit of finagling, so fortunately they haven't bricked everything.
And yeah staged updates or even just... some testing? Not sure how this one slipped through.
I'd bet my ass this was caused by terrible practices brought on by suits demanding more "efficient" releases.
"Why do we do so much testing before releases? Have we ever had any problems before? We're wasting so much time that I might not even be able to buy another yacht this year"
At least nothing like this happens in the airline industry
Certainly not! Or other industries for that matter. It's a good thing executives everywhere aren't just concentrating on squeezing the maximum amount of money out of their companies and funneling it to themselves and their buddies on the board.
Sure, let's "rightsize" the company by firing 20% of our workforce (but not management!) and raise prices 30%, and demand that the remaining employees maintain productivity at the level it used to be before we fucked things up. Oh and no raises for the plebs, we can't afford it. Maybe a pizza party? One slice per employee though.
One of my coworkers, while waiting on hold for 3+ hours with our company’s outsourced helpdesk, noticed after booting into safe mode that the Crowdstrike update had triggered a snapshot that she was able to roll back to and get back on her laptop. So at least that’s a potential solution.
Agreed, this will probably kill them over the next few years unless they can really magic up something.
They probably don't get sued - their contracts will have indemnity clauses against exactly this kind of thing, so unless they seriously misrepresented what their product does, this probably isn't a contract breach.
If you are running crowdstrike, it's probably because you have some regulatory obligations and an auditor to appease - you aren't going to be able to just turn it off overnight, but I'm sure there are going to be some pretty awkward meetings when it comes to contract renewals in the next year, and I can't imagine them seeing much growth
Nah. This has happened with every major corporate antivirus product. Multiple times. And the top IT people advising on purchasing decisions know this.
Yep. This is just uninformed people thinking this doesn't happen. It's been happening since av was born. It's not new and this will not kill CS they're still king.
At my old shop we still had people giving money to checkpoint and splunk, despite numerous problems and a huge cost, because they had favourites.
Don't most indemnity clauses have exceptions for gross negligence? Pushing out an update this destructive without it getting caught by any quality control checks sure seems grossly negligent.
Can't; the project manager ate all the crayons
Why is it bad to do on a Friday? Based on your last paragraph, I would have thought Friday is probably the best week day to do it.
Most companies, mine included, try to roll out updates during the middle or start of a week. That way if there are issues the full team is available to address them.
And hence the term read-only Friday.
Or someone selected "env2" instead of "env1" (#cattleNotPets names) and tested in prod by mistake.
Look, it's a gaffe and someone's fired. But it doesn't mean fuck ups are endemic.
Was it not possible for MS to design their safe mode to still “work” when Bitlocker was enabled? Seems strange.
I'm not sure what you'd expect to be able to do in a safe mode with no disk access.
I think you're on the nose, here. I laughed at the headline, but the more I read the more I see how fucked they are. Airlines. Industrial plants. Fucking governments. This one is big in a way that will likely get used as a case study.
The London Stock Exchange went down. They're fukd.
Yeah saw that several steel mills have been bricked by this, that's months and millions to restart
Got a link? I find it hard to believe that a process like that would stop because of a few windows machines not booting.
Those machines should be airgapped and no need to run Crowdstrike on them. If the process controller machines of a steel mill are connected to the internet and installing auto updates then there really is no hope for this world.
No, regulatory auditors have boxes that need checking, regardless of the reality of the technical infrastructure.
I work in an environment where the workstations aren't on the Internet there's a separate network, there's still a need for antivirus and we were hit with bsod yesterday
There is no unsafer place than isolated network. AV and xdr is not optional in industry/healthcare etc.
I don't know how to tell you this, but....
There are a lot of heavy manufacturing tools that are controlled and have their interface handled by Windows under the hood.
They're not all networked, and some are super old, but a more modernized facility could easily be using a more modern version of Windows and be networked to have flow of materials, etc more tightly integrated into their systems.
The higher precision your operation, the more useful having much more advanced logs, networked to a central system, becomes in tracking quality control.
Imagine if after the fact, you could track a set of .1% of batches that are failing more often and look at the per second logs of temperature they were at during the process, and see that there's 1° temperature variance between the 30th to 40th minute that wasn't experienced by the rest of your batches. (Obviously that's nonsense because I don't know anything about the actual process of steel manufacturing. But I do know that there's a lot of industrial manufacturing tooling that's an application on top of windows, and the higher precision your output needs to be, the more useful it is to have high quality data every step of the way.)
Testing in production will do that
Manglement is the good term lmao
What lawsuits do you think are going to happen?
They can have all the clauses they like but pulling something like this off requires a certain amount of gross negligence that they can almost certainly be held liable for.
Whatever you say my man. It's not like they go through very specific SLA conversations and negotiations to cover this or anything like that.
I forgot that only people you have agreements with can sue you. This is why Boeing hasn't been sued once recently for their own criminal negligence.
👌👍
😔💦🦅🥰🥳
Forget lawsuits, they're going to be in front of congress for this one
For what? At best it would be a hearing on the challenges of national security with industry.
Don't we blame MS at least as much? How does MS let an update like this push through their Windows Update system? How does an application update make the whole OS unable to boot? Blue screens on Windows have been around for decades, why don't we have a better recovery system?
Crowdstrike runs at ring 0, effectively as part of the kernel. Like a device driver. There are no safeguards at that level. Extreme testing and diligence is required, because these are the consequences for getting it wrong. This is entirely on crowdstrike.
This didn't go through Windows Update. It went through the ctowdstrike software directly.
The amount of servers running Windows out there is depressing to me
The four multinational corporations I worked at were almost entirely Windows servers with the exception of vendor specific stuff running Linux. Companies REALLY want that support clause in their infrastructure agreement.
I've worked as an IT architect at various companies in my career and you can definitely get support contracts for engineering support of RHEL, Ubuntu, SUSE, etc. That isn't the issue. The issue is that there are a lot of system administrators with "15 years experience in Linux" that have no real experience in Linux. They have experience googling for guides and tutorials while having cobbled together documents of doing various things without understanding what they are really doing.
I can't tell you how many times I've seen an enterprise patch their Linux solutions (if they patched them at all with some ridiculous rubberstamped PO&AM) manually without deploying a repo and updating the repo treating it as you would a WSUS. Hell, I'm pleasantly surprised if I see them joined to a Windows domain (a few times) or an LDAP (once but they didn't have a trust with the Domain Forest or use sudoer rules...sigh).
Reminds me of this guy I helped a few years ago. His name was Bob, and he was a sysadmin at a predominantly Windows company. The software I was supporting, however, only ran on Linux. So since Bob had been a UNIX admin back in the 80s they picked him to install the software.
But it had been 30 years since he ever touched a CLI. Every time I got on a call with him, I'd have to give him every keystroke one by one, all while listening to him complain about how much he hated it. After three or four calls I just gave up and used the screenshare to do everything myself.
AFAIK he's still the only Linux "sysadmin" there.
"googling answers", I feel personally violated.
/s
To be fare, there is not reason to memorize things that you need once or twice. Google is tool, and good for Linux issues. Why debug some issue for few hours, if you can Google resolution in minutes.
I'm not against using Google, stack exhange, man pages, apropos, tldr, etc. but if you're trying to advertise competence with a skillset but you can't do the basics and frankly it is still essentially a mystery to you then youre just being dishonest. Sure use all tools available to you though because that's a good thing to do.
Just because someone breathed air in the same space occasionally over the years where a tool exists does not mean that they can honestly say that those are years of experience with it on a resume or whatever.
Capitalism makes them to.
Agreed. If you are not incompetent, you will remember the stuff that you use often. You will know exactly where to look to refresh your memory for things you use infrequently, and when you do need to look something up, you will understand the solution and why it’s correct. Being good at looking things up, is like half the job.
RedHat, Ubuntu, SUSE - they all exist on support contracts.
I dunno, but doesn't like a quarter of the internet kinda run on Azure?
And 60% of Azure is running Linux lol
so 40% of azure crashes a quarter of the internet...
I guess Spotify was running on the other 40%, as many other services
Said another way, 3/4 of the internet isn't on Unsure cloud blah-blah.
And azure is - shhh - at least partially backed by Linux hosts. Didn't they buy an AWS clone and forcibly inject it with money like Bobby Brown on a date in the hopes of building AWS better than AWS like they did with nokia? MS could be more protectively diverse than many of its best customers.
I've had my PC shut down for updates three times now, while using it as a Jellyfin server from another room. And I've only been using it for this purpose for six months or so.
I can't imagine running anything critical on it.
Windows server, the OS, runs differently from desktop windows. So if you're using desktop windows and expecting it to run like a server, well, that's on you. However, I ran windows server 2016 and then 2019 for quite a few years just doing general homelab stuff and it is really a pain compared to Linux which I switched to on my server about a year ago. Server stuff is just way easier on Linux in my experience.
It doesn't have to, though. Linux manages to do both just fine, with relatively minor compromises.
Expecting an OS to handle keeping software running is not a big ask.
Yup, I use Linux to run a Jellyfin server, as well as a few others things. The only problem is that the CPU I'm using (Ryzen 1st gen) will crash every couple weeks or so (known hardware fault, I never bothered to RMA), but that's honestly not that bad since I can just walk over and restart it. Before that, it ran happily on an old Phenom II from 2009 for something like 10 years (old PC), and I mostly replaced it because the Ryzen uses a bit less electricity (enough that I used to turn the old PC off at night; this one runs 24/7 as is way more convenient).
So aside from this hardware issue, Linux has been extremely solid. I have a VPS that tunnels traffic into my Jellyfin and other services from outside, and it pretty much never goes down (I guess the host reboots it once a year or something for hardware maintenance). I run updates when I want to (when I remember, which is about monthly), and it only goes down for like 30 sec to reboot after updates are applied.
So yeah, Linux FTW, once it's set up, it just runs.
You can try to use watchdog to automatically restart on crashes. Or go through RMA.
I could, but it's a pretty rare nuisance. I'd rather just replace the CPU than go through RMA, a newer gen CPU is quite inexpensive, I could probably get by with a <$100 CPU since anything AM4 should work (I have an X370 with support for 5XXX series CPUs).
I'm personally looking at replacing it with a much lower power chip, like maybe something ARM. I just haven't found something that would fit well since I need 2-4 SATA (PCIe card could work), 16GB+ RAM, and a relatively strong CPU. I'm hopeful that with ARM Snapdragon chips making their way to laptops and RISC-V getting more available, I'll find something that'll fit that niche well. Otherwise, I'll just upgrade when my wife or I upgrade, which is what I usually do.
4 SATA, 8GB RAM is easy to find. What do you need 16 gigs for? Compiling Gentoo?
Star64 for ARM and Quartz64 for RV.
Off the car lot, we say 'request'. But good on you for changing careers.
I really have no idea why you think your choice of wording would be relevant to the discussion in any way, but OK...
Not judging, but why wouldn't you run Linux for a server?
Because I only have one PC (that I need for work), and I can't be arsed to cock around with dual boot just to watch movies. Especially when Windows will probably break that at some point.
Can you use Linux as main OS then? What do you need your computer to do?
I need to run windows software that makes other windows software, that will be run on our customers (who pay us quite well) PCs that also run windows.
Plus gaming. I'm not switching my primary box to Linux at any point. If I get a mini server, that will probably ruin Linux.
Mingw, but whatever. Maybe there is somethong mingw can't do.
Unless it is Apex and some other worst offenders or you use GPU from the only company actively hostile to linux, gaming is fine.
Wow dude you’re so cool. I bet that made you feel so superior. Everyone on here thinks you are so badass.
I do as well!
Wow and the most predictable reply too? Poor guy. Better luck next time.
Where did you think Microsoft was getting all (hyperbole) of their money from?
I know i was really surprised how many there are. But honestly think of how many companies are using active directory and azure
>Make a kernel-level antivirus
>Make it proprietary
>Don't test updates... for some reason??
never do updates on a Friday.
Excuse me, what now? I didn't get that memo.
Yeah it's great :-) 4 10hr shifts and every weekend is a 3 day weekend
Is the 4x10 really worth the extra day off? Tbh I'm not sure it would work very well for me... I find just one 10-hour day to be kinda draining, so doing that 4 times a week every week feels like it might just cancel out any benefits of the extra day off.
I am very used to it so I don't find it draining. I tried 5x8 once and it felt more like working an extra day than getting more time in the afternoon. If that makes sense. I also start early around 7am, so I am only staying a little later than other people
I changed jobs because the new management was all "if I can't look at your ass you don't work here" and I agreed.
I now work remotely 100% and it's in the union contract with the 21vacation days and 9x9 compressed time and regular raises. The view out my home office window is partially obscured by a floofy cat and we both like it that way.
I'd work here until I die.
Yep, anything done on Friday can enter the world on a Monday.
I don't really have any plans most weekends, but I sure as shit don't plan on spending it fixing Friday's fuckups.
And honestly, anything that can be done Monday is probably better done on Tuesday. Why start off your week by screwing stuff up?
We have a team policy to never do externally facing updates on Fridays, and we generally avoid Mondays as well unless it's urgent. Here's roughly what each day is for:
If things go sideways, we come in on Thu to straighten it out, but that almost never happens.
Actually I was not even joking. I also work in IT and have exactly the same opinion. Friday is for easy stuff!
You posted this 14 hours ago, which would have made it 4:30 am in Austin, Texas where Cloudstrike is based. You may have felt the effect on Friday, but it's extremely likely that the person who made the change did it late on a Thursday.
Never update unless something is broken.
This is fine as long as you politely ask everyone on the Internet to slow down and stop exploiting new vulnerabilities.
I think vulnerabilities found count as "something broken" and chap you replied to simply did not think that far ahead hahah
For real - A cyber security company should basically always be pushing out updates.
Exactly. You don't know what the vulnerabilities are, but the vendors pushing out updates typically do. So stay on top of updates to limit the attack surface.
Major releases can wait, security updates should be pushed as soon as they can be proven to not break prod.
Notes: Version bump: Eric is a twat so I removed his name from the listed coder team members on the about window.
git push --force
leans back in chair productive day, productive day indeed
git commit -am "Fixed" && git push --forceThat's advice so smart you're guaranteed to have massive security holes.
BTW, I use Arch.
If it was Arch you'd update once every 15 minutes whether anything's broken or not.
I use Tumbleweed, so I only get updates once/day, twice if something explodes. I used to use Arch, so my update cycle has lengthened from 1-2x/day to 1-2x/week, which is so much better.
gets two update notifications
Ah, must be explosion Wednesday
I really like the tumbleweed method, seems like the best compromise between arch and debian style updates.
I think a lot of what (open)SUSE does is pretty solid. For example, microOS is a fantastic compromise between a stable base and a rolling userspace, and I think a lot of people would do well to switch to it from Leap. I currently use Leap for my NAS, but I do plan to switch to microOS.
This is AV, and even possible that it is part of definitions (for example some windows file deleted as false positive). You update those daily.
Yeah my plans of going to sleep last night were thoroughly dashed as every single windows server across every datacenter I manage between two countries all cried out at the same time lmao
Here's the fix: (or rather workaround, released by CrowdStrike) 1)Boot to safe mode/recovery 2)Go to C:\Windows\System32\drivers\CrowdStrike 3)Delete the file matching "C-00000291*.sys" 4)Boot the system normally
It's disappointing that the fix is so easy to perform and yet it'll almost certainly keep a lot of infrastructure down for hours because a majority of people seem too scared to try to fix anything on their own machine (or aren't trusted to so they can't even if they know how)
They also gotta get the fix through a trusted channel and not randomly on the internet. (No offense to the person that gave the info, it’s maybe correct but you never know)
Yeah, and it's unknown if CS is active after the workaround or not (source: hackernews commentator)
True, but knowing what the fix might be means you can Google it and see what comes back. It was on StackOverflow for example, but at the time of this comment has been taken offline for moderation - whatever that means.
Meh. Even if it bricked crowdstrike instead of helping, you can just restore the file you deleted. A file in that folder can't brick a windows system.
Yeah and a lot of corpo VPNs are gonna be down from this too.
This sort of fix might not be accessible to a lot of employees who don't have admin access on their company laptops, and if the laptop can't be accessed remotely by IT then the options are very limited. Trying to walk a lot of nontechnical users through this over the phone won't go very well.
Yup, that's me. We booted into safe mode, tried navigating into the CrowdStrike folder and boom: permission denied.
Half our shit can't even boot into safe mode because it's encrypted and we don't have the keys rofl
If you don't have the keys, what the hell are you doing? We have bitlocker enabled and we have a way to get the recovery key so it's not a problem. Just a huge pain in the ass.
I went home lol. Some other poor schmucks are probably gonna reformat the computers.
Might seem easy to someone with a technical background. But the last thing businesses want to be doing is telling average end users to boot into safe mode and start deleting system files.
If that started happening en masse we would quickly end up with far more problems than we started with. Plenty of users would end up deleting system32 entirely or something else equally damaging.
I do IT for some stores. My team lead briefly suggested having store managers try to do this fix. I HARD vetoed that. That's only going to do more damage.
It might not even be that. A lot of places have many servers (and even more virtual servers) running crowdstrike. Some places also seem to have it on endpoints too.
That's a lot of machines to manually fix.
That is unfortunate but also leads me to a different question
Why do people like windows server? I've had to use it a couple of times for work and although it's certainly better than just using the desktop windows it's so heavy compared to running something like Debian
In our case, the fact we were using windows server actually made it a worse experience for customers aswell because the hardware was not up to it (because budget constraints) so it just chugged and slowed down everything making it a terrible experience for everyone involved (not to mention how often it'd have to be rebooted because a service wouldn't restart)
And people need to travel to remote machines to do this in person
You can do it over the phone. I just did a few dozen this morning and it was relatively easy.
yeah, sometimes that's just not an option...
I wouldn’t fix it if it’s not my responsibly at work. What if I mess up and break things further?
When things go wrong, best to just let people do the emergency process.
I'm on a bridge still while we wait for Bitlocker recovery keys, so we can actually boot into safemode, but the Bitkocker key server is down as well...
Gonna be a nice test of proper backups and disaster recovery protocols for some organisations
Chaos Monkey test
Man, it sure would suck if you could still get to safe mode from pressing f8. Can you imagine how terrible that'd be?
You hold down Shift while restarting or booting and you get a recovery menu. I don’t know why they changed this behaviour.
That was the dumbest thing to learn this morning.
A driver failure, yeesh. It always sucks to deal with it.
Not that easy when it's a fleet of servers in multiple remote data centers. Lots of IT folks will be spending their weekend sitting in data center cages.
This is going to be a Big Deal for a whole lot of people. I don't know all the companies and industries that use Crowdstrike but I might guess it will result in airline delays, banking outages, and hospital computer systems failing. Hopefully nobody gets hurt because of it.
Big chunk of New Zealands banks apparently run it, cos 3 of the big ones can't do credit card transactions right now
It was mayhem at PakNSave a bit ago.
In my experience it’s always mayhem at PakNSave.
If anything, it's probably calmed P'n'S down a bit...
Bitcoin still up and running perhaps people can use that
Bitcoin Cash maybe. Didn't they bork Bitcoin (Core) so you have to wait for confirmations in the next block?
Several 911 systems were affected or completely down too
CrowdStrike: It's Friday, let's throw it over the wall to production. See you all on Monday!
Wow, I didn't realize CrowdStrike was widespread enough to be a single point of failure for so much infrastructure. Lot of airports and hospitals offline.
Flights grounded in the US.
The System is Down
Ironic. They did what they are there to protect against. Fucking up everyone's shit
An offline server is a secure server!
Honestly my philosophy these days, when it comes to anything proprietary. They just can't keep their grubby little fingers off of working software.
At least this time it was an accident.
There is nothing unsafer than local networks.
AV/XDR is not optional even in offline networks. If you don't have visibility on your network, you are totally screwed.
The thought of a local computer being unable to boot because some remote server somewhere is unavailable makes me laugh and sad at the same time.
I don't think that's what's happening here. As far as I know it's an issue with a driver installed on the computers, not with anything trying to reach out to an external server. If that were the case you'd expect it to fail to boot any time you don't have an Internet connection.
Windows is bad but it's not that bad yet.
It’s just a fun coincidence that the azure outage was around the same time.
Yep, and it's harder to fix Windows VMs in Azure that are effected because you can't boot them into safe mode the same way you can with a physical machine.
Foof. Nightmare fuel.
So, like the UbiSoft umbilical but for OSes.
Edit: name of publisher not developer.
A remote server that you pay some serious money to that pushes a garbage driver that prevents yours from booting
Not only does it (possibly) prevent booting, but it will also bsod it first so you'll have to see how lucky you get.
Goddamn I hate crowdstrike. Between this and them fucking up and letting malware back into a system, I have nothing nice to say about them.
It's bsod on boot
And anything encrypted with bitlocker can't even go into safe mode to fix it
It doesn't consistently bsod on boot, about half of affected machines did in our environment, but all of them did experience a bsod while running. A good amount of ours just took the bad update, bsod'd and came back up.
yeah so you can't get Chinese government spyware installed.
Clownstrike
Yep, stuck at the airport currently. All flights grounded. All major grocery store chains and banks also impacted. Bad day to be a crowdstrike employee!
My flight was canceled. Luckily that was a partner airline. My actual airline rebooked me on a direct flight. Leaves 3 hours later and arrives earlier. Lower carbon footprint. So, except that I'm standing in queue so someone can inspect my documents it's basically a win for me. 😆
https://www.theregister.com/ has a series of articles on what's going on technically.
Latest advice...
Boot Windows into Safe Mode or WRE.
Go to C:\Windows\System32\drivers\CrowdStrike
Locate and delete file matching "C-00000291*.sys"
Boot normally.
Yep, this is the stupid timeline. Y2K happening to to the nuances of calendar systems might have sounded dumb at the time, but it doesn't now. Y2K happening because of some unknown contractor's YOLO Friday update definitely is.
My work PC is affected. Nice!
Plot twist: you're head of IT
Same! Got to log off early 😎
Dammit, hit us at 5pm on Friday in NZ
4:00PM here in Aus. Absolutely perfect for an early Friday knockoff.
Noice!
A few years ago when my org got the ask to deploy the CS agent in linux production servers and I also saw it getting deployed in thousands of windows and mac desktops all across, the first thought that came to mind was "massive single point of failure and security threat", as we were putting all the trust in a single relatively small company that will (has?) become the favorite target of all the bad actors across the planet. How long before it gets into trouble, either because if it's own doing or due to others?
I guess that we now know
No bad actors did this, and security goes in fads. Crowdstrike is king right now, just as McAfee/Trellix was in the past. If you want to run around without edr/xdr software be my guest.
I don't think anyone is saying that... But picking programs that your company has visibility into is a good idea. We use Wazuh. I get to control when updates are rolled out. It's not a massive shit show when the vendor rolls out the update globally without sufficient internal testing. I can stagger the rollout as I see fit.
You can do this with CS as well, but the dumbasses where pushing major updates via channel files which aren't for that. They tried to squeak by without putting out a major update via the sensor updates which you can control. Basically they fucked up their own structure because a bunch of people where complaining and more than likely management got involved and overwrote best practices.
Hmm. Is it safer to have a potentially exploitable agent running as root and listening on a port, than to not have EDR running on a well-secured low-churn enterprise OS - sit down, Ubuntu - adhering to best practice for least access and least-services and good role-sep?
It's a pickle. I'm gonna go with "maybe don't lock down your enterprise Linux hard and then open a yawning garage door of a hole right into it" but YMMV.
Reality is, if your users are educated, then your more secure than any edr with dumbass users. But we all know this is a pipe dream.
All of the security vendors do it over enough time. McAfee used to be the king of them.
https://www.zdnet.com/article/defective-mcafee-update-causes-worldwide-meltdown-of-xp-pcs/
https://www.bleepingcomputer.com/news/security/trend-micro-antivirus-modified-windows-registry-by-mistake-how-to-fix/
https://www.techradar.com/news/microsoft-releases-fix-for-botched-windows-defender-update-but-its-still-facing-problems
I'm so exhausted... This is madness. As a Linux user I've busy all day telling people with bricked PCs that Linux is better but there are just so many. It never ends. I think this is outage is going to keep me busy all weekend.
My dad needed a CT scan this evening and the local ER's system for reading the images was down. So they sent him via ambulance to a different hospital 40 miles away. Now I'm reading tonight that CrowdStrike may be to blame.
Honestly kind of excited for the company blogs to start spitting out their
disaster recoverycrisis management stories.I mean - this is just a giant test of
disaster recoverycrisis management plans. And while there are absolutely real-world consequences to this, the fix almost seems scriptable.If a company uses IPMI (
CalledBranded AMT and sometimes vPro by Intel), and their network is intact/the devices are on their network, they ought to be able to remotely address this.But that’s obviously predicated on them having already deployed/configured the tools.
Been at work since 5AM... finally finished deleting the C-00000291*.sys file in CrowdStrike directory.
182 machines total. Thankfully the process in of itself takes about 2-3 minutes. For virtual machines, it's a bit of a pain, at least in this org.
lmao I feel kinda bad for those companies that have 10k+ endpoints to do this to. Eff... that. Lot's of immediate short term contract hires for that, I imagine.
crowdstrike sent a corrupt file with a software update for windows servers. this caused a blue screen of death on all the windows servers globally for crowdstrike clients causing that blue screen of death. even people in my company. luckily i shut off my computer at the end of the day and missed the update. It's not an OTA fix. they have to go into every data center and manually fix all the computer servers. some of these severs have encryption. I see a very big lawsuit coming...
We had a bad CrowdStrike update years ago where their network scanning portion couldn’t handle a load of DNS queries on start up. When asked how we could switch to manual updates we were told that wasn’t possible. So we had to black hole the update endpoint via our firewall, which luckily was separate from their telemetry endpoint. When we were ready to update, we’d have FW rules allowing groups to update in batches. They since changed that but a lot of companies just hand control over to them. They have both a file system and network shim so it can basically intercept **everything **
My favourite thing has been watching sky news (UK) operate without graphics, trailers, adverts or autocue. Back to basics.
lol
too bad me posting this will bump the comment count though. maybe we should try to keep the vote count to 404
Linux and Mac just got free advertisment.
Annoyingly, my laptop seems to be working perfectly.
That's the burden when you run Arch, right?
lol he said it's working
He said it’s working annoyingly.
AWS No!!!
Oh wait it's not them for once.
One possible fix is to delete a particular file while booting in safe mode. But then they'll need to fix each system manually. My company encrypts the disks as well so it's going to be a even bigger pain (for them). I'm just happy my weekend started early.
Yeah that would be case in most laptops. So if bitlocker is involved as well what could be the possible fix.
That would be funny
Yeah, most large orgs have a key server, or back up to AD. If you don't have that, and no recovery key, you're fucked and that data is gone.
What if that is running crowdstrike?
I'll give you one guess.
(That's why when I was in charge of that stuff at one company, I had that recovery key printed out and kept separately in a lockbox.)
Even if they do, though, if your business isn't 20 people at 1 location, it's going to be a giant headache to get people everywhere to fix them all.
You have ta have access to boot in safe mode too, I guess I can't on my work pc for example.
What a shitty workaround & might crowd strike burn in hell lol
Enjoy your weekend unless you are in IT
Never trust a texan
Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half...
oh joy. can’t wait to have to fix this for all of our clients today…
I'm so tired of all the fun....
You’re going to have fun whether you like it or not.
(Seemed appropriate lol)
"Today", right. I wish you a good weekend stranger.
My company used to use something else but after getting hacked switched to crowdstrike and now this. Hilarious clownery going on. Fingers crossed I'll be working from home for a few days before anything is fixed.
Stop running production services on M$. There is a better backend OS.
This is a better article. It's a CrowdStrike issue with an update (security software)
I agree that's a better article, thanks for sharing
Np man. Thanks for mentioning it.
I see a lot of hate ITT on kernel-level EDRs, which I wouldn't say they deserve. Sure, for your own use, an AV is sufficient and you don't need an EDR, but they make a world of difference. I work in cybersecurity doing Red Teamings, so my job is mostly about bypassing such solutions and making malware/actions within the network that avoids being detected by it as much as possible, and ever since EDRs started getting popular, my job got several leagues harder.
The advantage of EDRs in comparison to AVs is that they can catch 0-days. AV will just look for signatures, a known pieces or snippets of malware code. EDR, on the other hand, looks for sequences of actions a process does, by scanning memory, logs and hooking syscalls. So, if for example you would make an entirely custom program that allocates memory as Read-Write-Execute, then load a crypto dll, unencrypt something into such memory, and then call a thread spawn syscall to spawn a thread on another process that runs it, and EDR would correlate such actions and get suspicious, while for regular AV, the code would probably look ok. Some EDRs even watch network packets and can catch suspicious communication, such as port scanning, large data extraction, or C2 communication.
Sure, in an ideal world, you would have users that never run malware, and network that is impenetrable. But you still get at avarage few % of people running random binaries that came from phishing attempts, or around 50% people that fall for vishing attacks in your company. Having an EDR increases your chances to avoid such attack almost exponentionally, and I would say that the advantage it gives to EDRs that they are kernel-level is well worth it.
I'm not defending CrowdStrike, they did mess up to the point where I bet that the amount of damages they caused worldwide is nowhere near the amount damages all cyberattacks they prevented would cause in total. But hating on kernel-level EDRs in general isn't warranted here.
Kernel-level anti-cheat, on the other hand, can go burn in hell, and I hope that something similar will eventually happen with one of them. Fuck kernel level anti-cheats.
I was quite surprised when I heard the news. I had been working for hours on my PC without any issues. It pays off not to use Windows.
No one bother to test before deploying to all machines? Nice move.
This outage is probably costing a significant portion of Crowd strike's market cap. They're an 80 billion dollar company but this is a multibillion outage.
Someone's getting fired for this. Massive process failures like this means that it should be some high level managers or the CTO going out.
They're already down ~9% today:
https://finance.yahoo.com/quote/CRWD/
So I think you're late to the party for puts. Smart money IMO is on a call for a rebound at this point. Perhaps smarter money is looking through companies that may have been overlooked that would be CrowdStrike customers and putting puts on them. The obvious players are airlines, but there could be a ton of smaller cap stocks that outsource their IT to them, like regional trains and whatnot.
Regardless, I don't gamble w/ options, so I'm staying out. I could probably find a deal, but I have a day job to get to with nearly 100% odds of getting paid.
Nice. The first comment is basically saying, "they're best in class, so they're worth the premium." And then the general, "you'll probably do better by doing the opposite of /r/wallstreetbets" wisdom.
So yeah, if I wanted to gamble, I'd be buying calls for a week or so out when everyone realizes that the recovery was relatively quick and CrowdStrike is still best in class and retained its customers. I think that's the most likely result here. Switching is expensive for companies like this, and the alternatives aren't nearly as good.
~20% down in the last month
They're about where they were back in early June. If they weather this, I don't see a reason why they wouldn't jump back to their all-time high in late June. This isn't a fundamental problem with the solution, it's a hiccup that, if they can recover quickly, will be just a blip like there was in early June.
I think it'll get hammered a little more today, and if the response looks good over the weekend, we could see a bump next week. It all depends on how they handle this fiasco this weekend.
YOLO 🚀🙈
Huh, so that's why the office couldn't order pizza last night lmfao
I picked the right week to be on PTO hahaha
This is proof you shouldn't invest everything in one technology. I won't say everyone should change to Linux because it isn't immune to this, but we need to push companies to support several OS
So that's why my work laptop is down for the count today. I'm even getting that same error as the thumbnail picture
If these affected systems are boot looping, how will they be fixed? Reinstall?
There is a fix people have found which requires manual booting into safe mode and removal of a file causing the BSODs. No clue if/how they are going to implement a fix remotely when the affected machines can't even boot.
Probably have to go old-skool and actually be at the machine.
And hope you are not using BitLocker cause then you are screwed since BitLocker is tied to CS.
You just need console access. Which if any of the affected servers are VMs, you’ll have.
Yes, VMs will be more manageable.
Exactly, and super fun when all your systems are remote!!!
It's not super awful as long as everything is virtual. It's annoying, but not painful like it would be for physical systems.
Really don't envy physical/desk side support folks today....
Do you have any source on this?
If you have an account you can view the support thread here: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19
Workaround Steps:
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
I can confirm it works after applying it to >100 servers :/
Nice work, friend. 🤝 [back pat]
It seems like it's in like half of the news stories.
It is possible to edit a folder name in windows drivers. But for IT departments that could be more work than a reimage
Having had to fix >100 machines today, I'm not sure how a reimage would be less work. Restoring from backups maybe, but reimage and reconfig is so painful
Yes, but there are less competent people. The main answer for any slightly complex issue at work is 'reimage' - the pancea to solve all problems. And reconfig of personal settings is the users problem.
It’s just one file to delete.
Everyone is assuming it’s some intern pushing a release out accidentally or a lack of QA but Microsoft also pushed out July security updates that have been causing bsods on the 9th(?). These aren’t optional either.
What’s the likelihood that the CS file was tested on devices that hadn’t got the latest windows security update and it was an unholy union of both those things that’s caused this meltdown. The timelines do potentially line up when you consider your average agile delivery cadence.
Apparently at work "some servers are experiencing problems". Sadly, none of the ones I need to use :(
A lot of people I work with were affected, I wasn't one of them. I had assumed it was because I put my machine to sleep yesterday (and every other day this week) and just woke it up after booting it. I assumed it was an on startup thing and that's why I didn't have it.
Our IT provider already broke EVERYTHING earlier this month when they remote installed" Nexthink Collector" which forced a 30+ minute CHKDSK on every boot for EVERYONE, until they rolled out a fix (which they were at least able to do remotely), and I didn't want to have to deal with that the week before I go in leave.
But it sounds like it even happened to running systems so now I don't know why I wasn't affected, unless it's a windows 10 only thing?
Our IT have had some grief lately, but at least they specified Intel 12th gen on our latest CAD machines, rather than 13th or 14th, so they've got at least one win.
Meanwhile Kaspersky: *thinks if so incompetent people can even make antivirus at all*
don't rely on one desktop OS too much. diversity is the best.
Dont rely on corpo trash at al.
Servers on Windows? Even domain controllers can be Linux-based.
Xfinity H&I network it down so I can't watch Star Trek. I get an error msg connection failure. Other channels work though.
And subscribed!
Interesting day
I legit have never been more happy to be unemployed.
This is the best summary I could come up with:
There are reports of IT outages affecting major institutions in Australia and internationally.
The ABC is experiencing a major network outage, along with several other media outlets.
Crowd-sourced website Downdetector is listing outages for Foxtel, National Australia Bank and Bendigo Bank.
Follow our live blog as we bring you the latest updates.
The original article contains 52 words, the summary contains 52 words. Saved 0%. I'm a bot and I'm open source!
Good bot!
Some intern is getting their ass beat right now, never release into prod without extensive test.
It's a fair point but I would rather diversify and also use something that is open / less opaque
ReactOS for the win!
Buy the dip!
But probably not immediately, probably slowly over time as contracts come due.
To whom? All their competitors have had incidents like this too.
Webroot had something similar ish earlier this year. Such a pain.
Probably because it's a Crowdstrike issue, they've pushed a bad update.
AFAICT Microsoft is busy placing ads on everything and screen logging user activity instead of making a resilient foundation.
For contrast: I've been running Fedora Atomic. I'm sure it is possible to add some kernel mod that completely breaks the system. But if there was a crash on boot, in most situations, I'd be able to roll back to the last working version of everything.
It's not just Windows, it's affecting services that people that primarily use other OS's rely on, like Outlook or Federated login.
In these situations, blame isn't a thing, because everyone knows that a LSE can happen to anyone at any time. The second you start to throw stones, people will throw them back when something inevitably goes wrong.
While I do fundamentally agree with you, and believe that the correct outcome should be "how do we improve things so that this never happens again", it's hard to attach blame to Microsoft when they're the ones that have to triage and ensure that communication is met.
banks wouldn't use something that black box. just trust me bro wouldn't be a good pitch
If you trust banks that much, I have very bad news for you.
Windows moment 🤣
This is why you create restore points if using windows.
I'm used to IT doing a lot of their work on the weekends as to not impact operations.
Bahaha 😂😂 continue using proprietary software, that's all you are going to get in addition to privacy issues... Switch to Linux.
Good ol microsloth
play stupid games win stupid prizes
Why do people run windows servers when Linux exists, it’s literally a no brainer.
This is a a ruse to make Work From Home end.
All IT people should go on general strike now.
It's Russia, or Iran or China or even our "ally" Saudi Arabia. So really, it's time to reset the clock to pre 1989. Cut Russia and China off completely, no investment, no internet, no students no tourist nothing. These people mean and are continually doing us harm and we still plod along and some unscrupulous types become agents for personal profit. Enough.
best day ever. the digitards get a wakeup call. how often have been lectured by imbeciles how great whatever dumbo closed source is. "i need photoshop", "windows powershell and i get work done", "azure and onedrive and teams...best shit ever", " go use NT, nobody will use a GNU".
yeah well, i hope every windows user would be kept of the interwebs for a year and mac users just burn in hell right away. lazy scum that justifies shitting on society for their own comfort. while everyone needs a drivers license, dumb fucking parents give tiktok to their kids...idiocracy will have a great election this winter.