Global internet outage - Disruption to airlines, banks, media and other businesses worldwide - July 19, 2024

[TootsieFootsie]

One of the stores that was affected.

The owner said that the issue for them wasn't that they didn't take cash, but that they may have to trust their customers more.
I wonder if they mean by keeping cash on the premises, or do some stores look with suspicion at being paid by cash.

 

[Niner]

I remember when the calendars were all about to flip to the year 2000, and the massive IT precautions and backups that had to be done in case the systems collapsed. Y2K it was called.

We put in tons of overtime hours at the place I was working, just to get ready. (Then nothing adverse happened. )

I too was scheduled to work at HP for this Y2K! But unfortunately, I had the worst flu I have ever had - 1999....

So - I guess I have no Crowdstrike thingy - because I am good to go here!

 

[ch_13]

The home affairs minister, Clare O’Neil, has just issued a statement about the outage. She says a fix has been issued but it may take a while – it’s understood every affected computer had to be rebooted.

"This is a technical issue, caused by a Crowdstrike update to its customers. They have issued a fix for this, allowing affected companies and organisations to reboot their systems without the problem."

Windows outage live: CrowdStrike has ‘issued a fix’ but problem may take some time to resolve, Clare O’Neil says

Blue screen of death error screens seen on Windows workstations across the globe. Follow live

www.theguardian.com

It's not just a straight reboot. Computers will have to be manually booted into safe mode first, then some files deleted before they can be restored. The standard way to do this means they have to go computer by computer, fixing each one manually.

Big firms may have some workarounds where they can use imaging and scripting to fix multiple machines at one time. But those have their own set of challenges and won't be easy to implement. From what I'm reading in the system admin forums, there's also an issue with the MS encryption tech, called BitLocker, where the key is needed to restore a machine, but the key is saved on a machine that needs to be restored. Catch-22.

This will not be a quick fix.

 

[SouthAussie]

It wasn't real smart for Crowdstrike to start their upgrade during the business day.

When I worked in telecomms, we used to start our upgrades late on a Saturday, and have Sunday free in case anything went belly up. Just to try to affect as few customers as possible, if there were problems.

In this 24-hour world someone will always be affected, but you try to limit the exposure of any problems.

imo

 

[Inthedetails]

One of the stores that was affected.

The owner said that the issue for them wasn't that they didn't take cash, but that they may have to trust their customers more.
I wonder if they mean by keeping cash on the premises, or do some stores look with suspicion at being paid by cash.

I think they mean they have to trust customers to return and pay.

FWIW, my city doesn't allow "no cash" businesses as it discriminates against populations who don't have credit/bank cards. (And, it helps to have the cash option on days like this.)

jmo

Edited to correct error: businesses in the city must accept cash. I originally work the exact opposite, as if my brain had it's own cyber outage.

 

[TootsieFootsie]

I may be affected, so far in only one way.
On my physical internet radio, two of the ABC stations haven't been able to connect for a few days.
So it's days ago, so may not be anything to do with what's happening now.

I have had this happen before, a couple of times when the races were on here.
I think the ABC may have needed the bandwidth back for broadcasts from here.

No races right now, but I am hoping this "fix" will get my stations back.

 

[nhmemorymaker]

I too was scheduled to work at HP for this Y2K! But unfortunately, I had the worst flu I have ever had - 1999....

So - I guess I have no Crowdstrike thingy - because I am good to go here!

Hewlett Packard????

 

[Niner]

Hewlett Packard????

yes! In Roseville, California.

 

[ch_13]

We put in tons of overtime hours at the place I was working, just to get ready. (Then nothing adverse happened. )

It's always been amusing to me how people don't see the connection between the time and effort put in to prevent Y2K and the lack of problems on that day. (Not you specifically SouthAussie, just making a general observation.)

An ounce of prevention may be worth a pound of cure, but humans don't really think that way.

We'll see it here too. I'm sure all this was foreseeable. But only after billions in damages (and perhaps even some lives lost with all the hospital systems down) will there be some kind of safety net put in place so a random software update doesn't result in all this havoc.

 

[TootsieFootsie]

I hope it's fixed by Saturday. I have some parcels to be picked up at Big W.
I see on their site they are having difficulties both online and instore.

 

[Inthedetails]

I hope it's fixed by Saturday. I have some parcels to be picked up at Big W.
I see on their site they are having difficulties both online and instore.

For me, the local impact (so far) is subway signs that let you know when a train is coming are out. But it's not super unusual for those to be out anyway. As of now, trains are still running and that's the main thing. Fingers crossed the system doesn't get snarled.

Just checked - and it looks like phones are out for NYC police and fire, but radio dispatch is working. Hope that doesn't cause problems.

How the global IT outage is impacting NYC services, transit, local airports

Basic New York City services are operating, including 911 and 311, but agencies are having difficulty accessing many of their functions controlled by Microsoft 365.

abc7ny.com

 

[SouthAussie]

It's always been amusing to me how people don't see the connection between the time put in to prevent Y2K and the lack of problems on that day. (Not you specifically SouthAussie, just making a general observation.)

An ounce of prevention may be worth a pound of cure, but humans don't really think that way.

We'll see it here too. I'm sure all this was foreseeable. But only after billions in damages (and perhaps even some lives lost with all the hospital systems down) will there be some kind of safety net put in place so a random software update doesn't result in all this havoc.

Most people aren't even aware of all the things that so many did to ensure operations could continue to run smoothly.

I was assigned to telecomms management for a medical facility at the time, at one point we even considered turning back the system clocks to avoid 2000 - until we could see the ramifications.
Instead we installed a lot of work-arounds.

 

[LeahBee]

We'll see it here too. I'm sure all this was foreseeable. But only after billions in damages (and perhaps even some lives lost with all the hospital systems down) will there be some kind of safety net put in place so a random software update doesn't result in all this havoc

SBM
God I hope there's no fatalities and this doesn't happen again.
This is giving me flashbacks to 2022 (my fellow Canadians will know) of a massive tech outage we had nation-wide.
We lost internet, cable, public transportation, government services, debit machines, traffic lights, fire alarms/sprinkler systems, and 911 services.. and someone did die. It took 2 days to resolve this little maintenance whoopsie... apparently they've put solutions in place finally (almost the same thing happened a year prior) but I guess we will just have to wait and see what happens when another software update goes wrong.

2022 Rogers Communications outage - Wikipedia

en.m.wikipedia.org

 

[TootsieFootsie]

I just read something about some fire alarms in Victoria may have stopped working.

 

[SouthAussie]

For me, the local impact (so far) is subway signs that let you know when a train is coming are out. But it's not super unusual for those to be out anyway. As of now, trains are still running and that's the main thing. Fingers crossed the system doesn't get snarled.

Just checked - and it looks like phones are out for NYC police and fire, but radio dispatch is working. Hope that doesn't cause problems.

How the global IT outage is impacting NYC services, transit, local airports

Basic New York City services are operating, including 911 and 311, but agencies are having difficulty accessing many of their functions controlled by Microsoft 365.

abc7ny.com

Luckily, our Triple 0 emergency system didn't go down in Australia. Those emergency systems are one of the first things to worry about, for sure.


“We understand Triple-0 services are currently not affected by this outage."

BREAKING: Websites across Australia go dark after apparent CrowdStrike update causes Windows crashes

Devices at the ABC, Foxtel, the Commonwealth Bank and many more have been impacted by blue screens of death on Windows PCs around the country.

www.cyberdaily.au

 

[Inthedetails]

It's always been amusing to me how people don't see the connection between the time and effort put in to prevent Y2K and the lack of problems on that day. (Not you specifically SouthAussie, just making a general observation.)

An ounce of prevention may be worth a pound of cure, but humans don't really think that way.

We'll see it here too. I'm sure all this was foreseeable. But only after billions in damages (and perhaps even some lives lost with all the hospital systems down) will there be some kind of safety net put in place so a random software update doesn't result in all this havoc.

That's the same with everything, though. People don't know how the sausage gets made. jmo

 

[Interested_But_Confused]

From what I can tell, it appears this isn't a problem *caused* by Microsoft or Windows. The third-party security software provider, Crowdstrike, seems to have released a software update *for* Windows, and that update has damaged the systems it was applied to.

Something like this is why a lot of system administrators for big companies wait before applying updates to the whole network. Updates should be tested in a sandbox environment where they can't do any real damage.

The irony is that these companies were trusting Crowdstrike to protect them. Never in their wildest dreams would they have expected Crowdstrike to be the problem.

 

[TootsieFootsie]

This is giving me flashbacks to 2022 (my fellow Canadians will know) of a massive tech outage we had nation-wide.

I remember that.
It was all over the news in Australia at the time.

 

[Interested_But_Confused]

It's not just a straight reboot. Computers will have to be manually booted into safe mode first, then some files deleted before they can be restored. The standard way to do this means they have to go computer by computer, fixing each one manually.

Big firms may have some workarounds where they can use imaging and scripting to fix multiple machines at one time. But those have their own set of challenges and won't be easy to implement. From what I'm reading in the system admin forums, there's also an issue with the MS encryption tech, called BitLocker, where the key is needed to restore a machine, but the key is saved on a machine that needs to be restored. Catch-22.

This will not be a quick fix.

I hate to say it, but systems using BitLocker may be completely screwed. The best they could probably hope for is reinstallation using the latest system backup; but for companies which require absolute real-time data, that won't be good enough.

Crowdstrike is going to be in big trouble. If nothing else, they'll have lost the trust companies had in them. And for a security company, trust is everything.

 

[Inthedetails]

Luckily, our Triple 0 emergency system didn't go down in Australia. Those emergency systems are one of the first things to worry about, for sure.


“We understand Triple-0 services are currently not affected by this outage."

BREAKING: Websites across Australia go dark after apparent CrowdStrike update causes Windows crashes

Devices at the ABC, Foxtel, the Commonwealth Bank and many more have been impacted by blue screens of death on Windows PCs around the country.

www.cyberdaily.au

Yeah, it looks like 911 emergency service still functions here as well as water system and traffic lights (and the stock market). Those critical operations apparently have their own systems and weren't affected by this update problem.

jmo