Global IT Outage reported. (AKA 'Old lags reminiscing)!

[Neilc28]

No, it is not MSs fault directly, but it was delivered through their update system, and that has caused many headaches in the past.

It also allowed FTDI to issue a driver update through microsoft updates that bricked and clone FTDI interfaces.
They attacked the innocent buyers, a move that was frowned on by MS and many people considered unethical.

However, it is possible for organisations to control updates to client machines with their servers, but I get the impressions this was a server update. Even then, they can be delayed, but that would involve extra time staff and money.

I'm sure there will be action behind the scenes though

The updates were controlled by Crowdstrikes agent on the hosts and not Windows Update, this is why people use Crowdstrike in the first place as content updates for CS can be deployed immediately minimising risk of day-zero exploits and attacks, and whilst I’d agree that extra resource (not huge) is required for managing the updates internally via SCCM, EM, WUfB or WSUS the reason people choose CS for that risk assurance and quality checking, not manageability.

How many cockups have we seen with day 1 iOS updates, and those don’t include other vendors updates either?

 

[Sangoma]

The updates were controlled by Crowdstrikes agent on the hosts and not Windows Update, this is why people use Crowdstrike in the first place as content updates for CS can be deployed immediately minimising risk of day-zero exploits and attacks, and whilst I’d agree that extra resource (not huge) is required for managing the updates internally via SCCM, EM, WUfB or WSUS the reason people choose CS for that risk assurance and quality checking, not manageability.

How many cockups have we seen with day 1 iOS updates, and those don’t include other vendors updates either?

OK, then it was nothing to do with MS

Don't know about iOS, since I stopped working, I have not used Apple (one of the best things about my no-choice retirement )

 

[AndrewFlannigan]

The problem is that alternatives for some key elements are either garbage, don’t have mainstream support, or simply don’t exist.

I cannot believe that's accurate.

The market for enterprise software is currently reported to be $5,000 billion - more than enough to attract new entrants on a daily basis. More probably there are a number of very cosy relationships which are anti-competitive and preventing new entrants from coming forward. For the last few years too many corrupt governments have deliberately reigned back anti-monopoly legislation and spending, and this latest chaos is just one of its kind.

Forcing organisations to diversify their IT is simply a bad idea,

Diversification is never a bad idea, as the Potato Famine example so amply proves.

...let’s not overlook that this is fundamentally a Crowdstrike kernel driver screw-up, and not directly Microsoft at fault (the public are unlikely to see it this way)

I rather think that Microsoft's questionable business practices and market dominance is definitely a major part of the problem. One political action, which could help prevent repetition, would be to take away commercial protection of their interface definitions, so that competing operating systems could be developed. In the past, there have been several clones of the Microsoft OSes that allowed existing applications to run but Microsoft have always found ways to discourage them. Requiring Microsoft to publish detailed OS interface specifications, would result in lots of argument but should reduce the chance of a similar problem in future, as alternative software is developed.

 

[Neilc28]

I cannot believe that's accurate.

The market for enterprise software is currently reported to be $5,000 billion - more than enough to attract new entrants on a daily basis. More probably there are a number of very cosy relationships which are anti-competitive and preventing new entrants from coming forward. For the last few years too many corrupt governments have deliberately reigned back anti-monopoly legislation and spending, and this latest chaos is just one of its kind.

So where’s the alternative for Active Directory/Entra? Sure there’s other LDAP systems out there, are they a competitor? Not even close imo. I said key systems not everything, as there are good solutions out there, but there are certain solution categories with realistically no competition.

Diversification is never a bad idea, as the Potato Famine example so amply proves.

I said “forcing” wasn’t a good idea, potato-related or not.

I rather think that Microsoft's questionable business practices and market dominance is definitely a major part of the problem. One political action, which could help prevent repetition, would be to take away commercial protection of their interface definitions, so that competing operating systems could be developed. In the past, there have been several clones of the Microsoft OSes that allowed existing applications to run but Microsoft have always found ways to discourage them. Requiring Microsoft to publish detailed OS interface specifications, would result in lots of argument but should reduce the chance of a similar problem in future, as alternative software is developed.

Questionable business practices are absolutely not unique to MS, and I would say they are actually more open and fair than many of the other big tech companies, they play on a level field and if you want to regulate them further then every other company must also adhere.

 

[Nod]

If a failure to prepare is a preparation for failure, what does that make a failure to prepare for failure?

Far too much reliance on computers these days IMO.

 

[Box Brownie]

If a failure to prepare is a preparation for failure, what does that make a failure to prepare for failure?

Far too much reliance on computers these days IMO.

For many......and in increasing numbers.....the computer is "the magic box" that does what I tell it.

IIRC there was some mention and/or discussion about MS Windows 12, when it appears, will be cloud based. Such a central server based setup originally meant that the individual computers(?) were just terminals. Again IIRC a thin client setup though no doubt would be way more sophisticated in the 21stC compared to 'back then'.

So, should that be the case......future such failures will be even more "back to pre computer pen & paper"

 

[lindsay]

If Windows 12 became cloud-based, we are back to the Chromebook way of working, which would surely lose MS loads of business? I could see a delegated cloud OS running on Azure platforms so that companies could create their own virtual desktops though.....hang on, that's Citrix/VMWare style working. Not for me thanks, I can live with MS365 just, but I'm going to be replacing Adobe CC with a DXO/Affinity combo to get away from subs.

 

[AndrewFlannigan]

If Windows 12 became cloud-based, we are back to the Chromebook way of working, which would surely lose MS loads of business? I could see a delegated cloud OS running on Azure platforms so that companies could create their own virtual desktops though.....hang on, that's Citrix/VMWare style working. Not for me thanks, I can live with MS365 just, but I'm going to be replacing Adobe CC with a DXO/Affinity combo to get away from subs.

I remember being sent to Hammersmith, to see a Citrix demo running on a HP Mini.

This would be a bit less than thirty years ago. There were about a dozen PCs attached. After half an hour, they managed to get two of the PCs logged in. The demonstration showed that, with luck and a following wind, you could run Windows on a server, at rather less than half the speed it would run locally, but only if you could get the connection to work!

My report was, shall we say, less than glowing...

 

[JonathanRyan]

If Windows 12 became cloud-based, we are back to the Chromebook way of working, which would surely lose MS loads of business?

Is Windows big business for MS any more? I thought most of their money came from Azure. Just like Amazon don't really sell stuff - that's a loss making hobby.

 

[Sangoma]

If Windows 12 became cloud-based, we are back to the Chromebook way of working, which would surely lose MS loads of business? I could see a delegated cloud OS running on Azure platforms so that companies could create their own virtual desktops though.....hang on, that's Citrix/VMWare style working. Not for me thanks, I can live with MS365 just, but I'm going to be replacing Adobe CC with a DXO/Affinity combo to get away from subs.

I think they would lose that way.
If Affinity was ported to Linux, I would ditch windows tomorrow.

I ran Linux for years, at work and at home, and the only issue was lack of software, but if windows goes cloudbased, it might just be the prompt needed for more Linux software.

 

[Neilc28]

I think they would lose that way.
If Affinity was ported to Linux, I would ditch windows tomorrow.

I ran Linux for years, at work and at home, and the only issue was lack of software, but if windows goes cloudbased, it might just be the prompt needed for more Linux software.

It won’t be fully cloud, it’ll be Windows As A Service, installed locally, but also streamable if needed, with certain features such as AI offloaded to the cloud to save on compute requirements. This helps to ensure feature-parity irrespective of where and when the platform is used.

Microsoft seminars have been quite transparent in detailing the future of the Windows platform, and for enterprises this can be a big deal as it has the potential of saving large amounts of money on hardware which can be obsolete in a few years.

Am I convinced? I’ve had enough decades of Citrix farms and RDS behind me to be sceptical, but there’s been huge advancements in communication speeds and GPU optimisation to make me think there may be something in it

 

[Sangoma]

Is Windows big business for MS any more? I thought most of their money came from Azure. Just like Amazon don't really sell stuff - that's a loss making hobby.

Good question.
In 1999 I changed the city hall from Novell to NT and Linux servers.
I was asked at a meeting why Linux, as they (The IS department whose toes I was stepping on) considered it wouldn't be around long.
I said then it would be around as long as people have to pay for windows, if MS wanted to kill it, they would have to make windows free

It seems they have just about done that now, and by "persuading" everyone to go to W10, have boosted the profitability of their other products

 

[lindsay]

SAAS is all very well, but it makes the WAN the single point of failure for a whole company or organisation, in an age of increasing cyber attacks from global actors. Never mind the dependence of the Internet backbone on subsea cables whose vulnerability has also recently been demonstrated. The IT megaliths (MS, Google, etc) may be good at marketing the downstream cost efficiencies, but are less vocal on the corresponding risks.

 

[Jelster]

That's Agile isn't it? Invented in the mid-90's, the methodology that you deliver to live whatever has been developed by a certain date, and test in production. Obviously it doesn't say that on the tin, but it's what happens. It's actually HMG policy now "Agile First". I am very familiar with a certain govt dept that does a software release EVERY THURSDAY on a critical system, and at least one in every three releases introduces new, often critical, bugs to the service offered, despite having a large test team in place. The developers decide what tests to do because they write the "user stories" instead of actual users or real business analysts.

Outsourcing QA to India? Yes, I've worked on two projects where the client did that. In both cases I had to build a team to do the real testing after the outsource company had run the automated tests that they had created. We always found major bugs that needed fixing, but often the software was already in production.

My son works for a software company, his role is sorting and reproducing customer issues. He hates updates, he knows he'll have a couple of long days following the roll out......

 

[JonathanRyan]

Good question.
In 1999 I changed the city hall from Novell to NT and Linux servers.
I was asked at a meeting why Linux, as they (The IS department whose toes I was stepping on) considered it wouldn't be around long.
I said then it would be around as long as people have to pay for windows, if MS wanted to kill it, they would have to make windows free

It seems they have just about done that now, and by "persuading" everyone to go to W10, have boosted the profitability of their other products

I looked it up. In turnover, Microsoft generate about twice as much from Windows as they do from LinkedIn. It's not quite a rounding error but it's not close to core business. Meanwhile Amazon still don't make a profit from selling things.

 

[AndrewFlannigan]

SAAS is all very well, but it makes the WAN the single point of failure for a whole company or organisation, in an age of increasing cyber attacks from global actors. Never mind the dependence of the Internet backbone on subsea cables whose vulnerability has also recently been demonstrated. The IT megaliths (MS, Google, etc) may be good at marketing the downstream cost efficiencies, but are less vocal on the corresponding risks.

I spent the last few years, before I retired from the fray, alternating between banks and telecoms.

I found it interesting how these organisations seemed, at the time, to be pulling back from blades and distributed processing and were extending the life of their mainframe systems. I found myself opening the TSO manuals (and sometimes the COBOL books) surprisingly often, having spent the previous decades learning the latest wonder languages!

 

[lindsay]

I spent a number of years developing systems and supporting them on Tandem Non-Stop systems. I've not yet seen anything to match the resilience of the Tandems, nor the speed for distributed processing, and rue the day HP bought the company. However I agree that things seemed much more efficient and reliable in the mainframe days (I worked first on ICL VME/B and COBOL, then IBM s360 TSO, COBOL/REXX/other minor languages and later tested the Police National Computer running on Burroughs BS2000/Adabas/Natural).

 

[JonathanRyan]

I spent a number of years developing systems and supporting them on Tandem Non-Stop systems. I've not yet seen anything to match the resilience of the Tandems, nor the speed for distributed processing, and rue the day HP bought the company. However I agree that things seemed much more efficient and reliable in the mainframe days (I worked first on ICL VME/B and COBOL, then IBM s360 TSO, COBOL/REXX/other minor languages and later tested the Police National Computer running on Burroughs BS2000/Adabas/Natural).

Cobol's paying like £750 a day at the moment....

 

[lindsay]

Jeez, I used to teach COBOL (but ANSI74, there are newer standards now) to RAF & civilians in the MoD!

 

[AndrewFlannigan]

I spent a number of years developing systems and supporting them on Tandem Non-Stop systems. I've not yet seen anything to match the resilience of the Tandems, nor the speed for distributed processing, and rue the day HP bought the company. However I agree that things seemed much more efficient and reliable in the mainframe days (I worked first on ICL VME/B and COBOL, then IBM s360 TSO, COBOL/REXX/other minor languages and later tested the Police National Computer running on Burroughs BS2000/Adabas/Natural).

We had a Tandem for one of the projects I worked on at BT.

My job was project quality control, so I spent a lot of time checking code against results, which was hardly groundbreaking but essential. My group included one of those programming teams with one superb coder, two competent coders and five coders I had to watch like a hawk. On the other hand, I generally got to leave at 5pm, which wasn't the case on many other jobs.

 

[AndrewFlannigan]

Cobol's paying like £750 a day at the moment....

According to the Bank of England inflation calculator, that's about the same as I was getting in the mid-90s.

 

[Harlequin565]

I spent a number of years developing systems and supporting them on Tandem Non-Stop systems. I've not yet seen anything to match the resilience of the Tandems, nor the speed for distributed processing, and rue the day HP bought the company.

Completely agree! I was 3rd line support on Compaq owned Tandem machines for Cellnet. When HP bought it, things went downhill. As my boss told me when I joined - you need an operator for Tandem systems and a dog. The dog is there to bite the operator if his/her hands go near the keyboard. Learned this the hard way when I took down all the SS7 links to Node A, then powered down Node B for the upgrade. We had 2 nodes.... Wasn't very non-stop that night...

 

[lindsay]

I used to love destructive resilience testing of the Tandem systems at Lloyds Bank - I'd stop one device at a time across the network, watching processes move to other cpu's, other disks taking over primary, gradually grinding the system down until the Ops Desk told me to stop making their life hell with all the alerts! It was even better when we installed NextStep machines to monitor the systems and automate recovery actions.

Sorry for thread creep.

 

[AndrewFlannigan]

Completely agree! I was 3rd line support on Compaq owned Tandem machines for Cellnet.

I left BT Swindon on a Friday and started at Cellnet/O2 Slough on the following Monday.

I was on one of the marketing/billing systems that was going along with Cellnet/O2 in the split and I was the only spare person who knew it, so it was a no brainer for them to take me. Courtesy of my BT boss, I was allowed to use the Swindon car park, so I could cross the bridge and catch the train to Reading, swap to the Slough train and walk out of Slough station and into the Front door of the O2 building.

I just looked at Google Maps and it's all gone now

 

[JonathanRyan]

According to the Bank of England inflation calculator, that's about the same as I was getting in the mid-90s.

Ooof. Yeah, I just checked that calculator and based on what they paid me then, £750 isn't nearly enough Mind you, I had to pretend I also knew CICS.

Also, I remember why I stopped writing Cobol.......

Meanwhile it seems Cloudstrike may be better at writing contracts than they are at software. Tech sites are reporting that any compensation may be limited to a discount on fees for the period of non service. I.e. most customers will receive a maximum of 1/31 of their monthly fee back.

 

[Plain Nev]

Back to the abacus and good old pen and paper, I say.

 

[Mr Bump]

I worked for HP in Erskine for a very high profile UK GOV account
we tested every update in about 3 sand boxes before live
the cost was horrendous though
i suspect all that happened with this one was cloudstrike dropped the ball and cut a few to many corners
interesting it was nothing to do with MS I find there products just get better so much desktop OS reliability

 

[AndrewFlannigan]

I just looked at Google Maps and it's all gone now

...but not forgotten...



I found this in my old files, a grab shot from the train, probably as it was just pulling out, which shows "The Big O". Taken on film and possibly with an Olympus LT1, the camera infamous for its cheesy overall brown leather covering.

 

[ecoleman]

What really happened

View: https://www.youtube.com/watch?v=wAzEJxOo1ts

 

[lindsay]

That's really interesting @ecoleman , a good techie explanation which also took me back to debugging from sysdumps on mainframes. Happy days! Clearly some shortcuts taken by Crowdstrike then, to get the updates out very quickly but without adequate code quality in the process reading the update files.

 

[AndrewFlannigan]

That's really interesting @ecoleman , a good techie explanation

I agree.

It's always good to hear from someone who knows what they're talking about!

 

[ChrisR]

Perhaps it indicates how old I am, but the very thought that mission-critical systems are running on Windows machines (other than as end-user access points) fills me with horror!

 

[Jakey_Boy]

Looks like there may have been some signs of things to come. Not just Windows potentially vulnerable.

Jake

CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes

 

[lindsay]

An interesting article - as ever The Register is on the money.
This is why greybeards like me (maybe not all of us though?) have reservations about Platform as a Service and Software as a Service, for the vulnerability it brings with no control over your own destiny. I guess it's hard to stop thinking of the days when everyone had their own computers and IT departments staffed with programmers, analysts and tech support; there were still crashes, but you had the staff to deal with it, and no or much less dependence on external resources and providers. Of course, we also didn't have the Internet, because the limited access to JANET and corporate WAN's made us safer from external hacking (many of us used bespoke ethernet rather than Netware), so we didn't need things like Crowdstrike etc, booting in kernel mode or from BIOS. Maybe my nostalgia is getting the better of me though.

 

[AndrewFlannigan]

Maybe my nostalgia is getting the better of me though.

We all need to be wary of this, of course, but I don't think you are in this case. I think, rather, that it's a general problem when large organisations hand over major systems to outsiders.

I've been working my way through Mr Justice Fraser's Judgement No.6 on "Horizon Issues" in Bates and the Post Office. Although I've only go to section 194 (of 452) it's painfully obvious that the Post Office, having outsourced the system, had very little understanding of what was going on. If they had their own staff designing and implementing the system, the outcome would, I'm sure, have been far different and a great deal less painfull for all concerned.

Of course, it has long been unrealistic for even the biggest companies to follow in the footsteps of Lyons Teashops and build their own computer. However, if you buy a computer and operating system, but have your own in house team do the applications, you have far more control. Designers and programmers who share the canteen with the shop floor are far more likely to know what your business needs and what it doesn't need.

 

[Faldrax]

An interesting article - as ever The Register is on the money.
This is why greybeards like me (maybe not all of us though?) have reservations about Platform as a Service and Software as a Service, for the vulnerability it brings with no control over your own destiny. I guess it's hard to stop thinking of the days when everyone had their own computers and IT departments staffed with programmers, analysts and tech support; there were still crashes, but you had the staff to deal with it, and no or much less dependence on external resources and providers. Of course, we also didn't have the Internet, because the limited access to JANET and corporate WAN's made us safer from external hacking (many of us used bespoke ethernet rather than Netware), so we didn't need things like Crowdstrike etc, booting in kernel mode or from BIOS. Maybe my nostalgia is getting the better of me though.

The company I work for as a Software engineer does have it's own PC's, IT Department, etc. - but the business global, so everything is hosted using cloud services - we write the software, but are still reliant on someone else's infrastructure for everything to work.
It was the same to some extent in my first commercial software role (many years ago) - but then it was the dial-up Tradacoms network, which provided a secure messaging hub which the major UK supermarkets uploaded order files to, and we then called up to download those orders - an 'electronic postbox'.
The one thing that hasn't changed is the drive to reduce costs - and many businesses are prepared to take the small risk (but with potentially huge consequences) that a big provider will fail if the saving is perceived to be great enough.

 

[Box Brownie]

We all need to be wary of this, of course, but I don't think you are in this case. I think, rather, that it's a general problem when large organisations hand over major systems to outsiders.

I've been working my way through Mr Justice Fraser's Judgement No.6 on "Horizon Issues" in Bates and the Post Office. Although I've only go to section 194 (of 452) it's painfully obvious that the Post Office, having outsourced the system, had very little understanding of what was going on. If they had their own staff designing and implementing the system, the outcome would, I'm sure, have been far different and a great deal less painfull for all concerned.

Of course, it has long been unrealistic for even the biggest companies to follow in the footsteps of Lyons Teashops and build their own computer. However, if you buy a computer and operating system, but have your own in house team do the applications, you have far more control. Designers and programmers who share the canteen with the shop floor are far more likely to know what your business needs and what it doesn't need.

Re: Lyons Teashop computer system..,....

I did an evening class on Data Processing & Computing. The woman lecturer/tutor who took the class (if I recall) was one of the lead program developers of LEO (Lyons Electronic Office) sadly I can't remember her name......but the class was IIRC around the early 70's.

 

[SsSsSsSsSnake]

Sure it wasn’t Fuji releasing a continuous focus update?

 

[lindsay]

The one thing that hasn't changed is the drive to reduce costs - and many businesses are prepared to take the small risk (but with potentially huge consequences) that a big provider will fail if the saving is perceived to be great enough.

This is the crux of it I agree. Of course companies generally don't understand and di risk management at all well, in particular not realising that the cost associated with a risk actualising can be put onto the balance sheet if they do it right, in the same way as opportunity -= "goodwill" - is recorded as an asset.

 

[Boots]

Tradacoms network

There's a name I never expected to see again!