British Airways IT Meltdown

[Saracen]

Is it just me, or does this seem .... weird?

I mean, besides the obvious inconvenience, there's just something about what's happened, and the "story" BA are putting out, together with the abysmal PR handling of it that just doesn't ring true.

Is there more going on than they're letting on?

If there is, what? Credible terrorism threat? WannaCry ransomware breakout? Invasion by Martians?

It just doesn't quite ring true.

Now, where'd I put my second-favourite tinfoil hat?

[Phage]

I had a similar thought. Whilst airline systems are notoriously old and cranky, I am amazed that ALL of their systems have hone down across the globe simultaneously. Some serious lessons there.

I fund see that Capita had a massive outage at the same time. Possibly related ?

[peterb]

It struck me as odd too. The story of power failure doesn't ring true - the only credible explanation for that would be a major power supply failure to a data centre, so the servers were down. That then raises the question of why there wasn't a secondary system mirrored system for something so mission critical, or a generator system to maintain the data centre in the event of a power supply failure.

It has been reported that BA have outsourced their IT infrastructure to India. One of the big companies there are Tata industries (who also own one of the larger global tier 1 fibre networks) so if a data centre went down, it wouldn't just be BA that was affected. However, noting Phage's comment about Capita, maybe it was a data centre failure?

Edit: just read this https://www.theregister.co.uk/2017/0...a_data_centre/

That definitely was a data centre failure - where the standby generators failed to start - but one in this country. Coincidences do happen, but it still seems odd.

( The Register also reported this https://www.theregister.co.uk/2017/0...shoring_plans/ in March)


Or maybe we are getting twitchy and looking for connections when they really were just two coincidental random events.

[Corky34]

I would imagine it effected the whole of BA because everything connects to one of Tata Consultancy Services data centers in India.

Ninja edit, beaten to it by peterb.

[snedger]

IT departments periodically carry out disaster recovery tests - where supposedly, upon system failure, the IT Infrastructure can be up and running at backup site(s) within a very short period of time.
Personally every disaster recovery test that I have witnessed has been a box ticking jokey waste of a time that in no way proved that that DR contingency was adequate,or even workable at all
Companies can gamble on saving money for years with farcical non working DR, but if the DR plans, when needed in an emergency, fail, then those savings are swallowed up very quickly with down time losses.
It sickens me that people will have had bonuses paid partly because of DR test goal achievement. "success" .
On a personal note, I'm hoping that a very long haul BA flight home from South America is not mucked up - the last few days of experiencing cock up on cock up has made BA look the paragon of efficiency - I do not want to get stuck here - yes, I really would prefer to get back to work

[Jonj1611]

"One theory of the IT expert, who does not wish to be named, is that when the power came back on the systems were unusable because the data was unsynchronised."

Quote from BBC

[DanceswithUnix]

"One theory of the IT expert, who does not wish to be named, is that when the power came back on the systems were unusable because the data was unsynchronised."

Quote from BBC

Makes no sense to me. How the heck can enterprise data be "unsynchronised", this isn't some consumer RAID 1 stripeset with NTFS doing a diskchk, or at least I hope it isn't. Let's hope they maintain their planes better than their data.

[Phage]

The CEO is refusing to step down - apparently having a catastrophic failure that puts you out of business for at least a day is just fine....

[Corky34]

I wouldn't be surprised if the CEO got his normal bonus, i can't remember the last time a CEO stepped down after a publicised major snafu.

[snedger]

Makes no sense to me. How the heck can enterprise data be "unsynchronised", this isn't some consumer RAID 1 stripeset with NTFS doing a diskchk, or at least I hope it isn't. Let's hope they maintain their planes better than their data.

No, it's a proper computer - mainframe - 3390 disk packs, not toy desktop computers

[ik9000]

The CEO is refusing to step down - apparently having a catastrophic failure that puts you out of business for at least a day is just fine....

this is the same einstein who has been ditching in flight snacks and drinks on short haul flights, ditching choosing a seat without paying (hand luggage only), wants to ditch included meals on long haul flights etc etc. IMO the best thing that could happen to BA would be to get rid of this chap and the decision makers at the top. They need some serious nous up there to turn things around, and from what I can see the current crop are not going to provide it. Mind you it all started going downhill when they merged/subsumed into Iberian Airways, but that's one for another time.

[Saracen]

The CEO is refusing to step down - apparently having a catastrophic failure that puts you out of business for at least a day is just fine....

Hmmm. We'll see.

A but like the old, cynical Yes-Minister-ish line, never believe any rumour until it's been officially denied, I'm a firm believer in the principle that if he has to refuse to step down, he's under serious threat, and the clock is ticking. And knives will be out.

Tick, tock, tick, tock ....

[Biscuit]

Well I'm glad not the only one who finds the timing oddly suspicious. My girlfriend (who doesn't usually think like this AT ALL) also pointed out the unusual timing.

Definitely something fishy going on.

No, it's a proper computer - mainframe - 3390 disk packs, not toy desktop computers

Sure, but at the same time if a system is THAT sophisticated, you would expect protection and synchronization technologies at least in-line with that of none "toy" computer systems. If data goes "out of sync", it shouldn't take the whole system down, it should just give some support staff some overtime and an overnight headache.

[snedger]

Well I'm glad not the only one who finds the timing oddly suspicious. My girlfriend (who doesn't usually think like this AT ALL) also pointed out the unusual timing.

Definitely something fishy going on.



Sure, but at the same time if a system is THAT sophisticated, you would expect protection and synchronization technologies at least in-line with that of none "toy" computer systems. If data goes "out of sync", it shouldn't take the whole system down, it should just give some support staff some overtime and an overnight headache.

The support staff are only as good as their experience and in depth system knowledge accrued from years of bodging and fire fighting.
Could be something as simple as PTFs {patches) were applied to the mainframe in real time, but not saved away for the next IPL - so post power failure the machine boots up minus critical software changes.
Something that is part of lessons learned for experienced IBMers, but completely new for an Indian software bureau - you just need to read some of the IBM mainframe technical forum posts to see that some Indian software houses employ people straight out of uni, give them no training and expect them to learn on the job - which they do, but critical mistakes can be made on the way to becoming experienced and there are no system experienced peers to guide them.
Have seen a number of outsourcing exercises where UK people are expected to pass on their knowledge before being made redundant.
A very painful experience - have seen people get so cynical and jaded, that in the end the UK people become completely un-proactive and only answer questions that are asked - the incoming outsourcees of course have no experience or knowledge of what questions to ask - as part of naive cost cutting, the UK staff are released far too soon - outsourcing declared a wonderful on paper success, bonuses paid - but then,wait a minute BOOM!
Some management have no appreciation of the value of system knowledge and experience. Staff for years, fire fight problems, saving companies thousand or millions of pounds, before being rewarded by being given the boot - admittedly some of them knowledge hoard, thinking that they are indispensable - no one is indispensable and that behaviour is just as scuzzy as for bonus chasing managers IMO.

[Phage]

Well I'm glad not the only one who finds the timing oddly suspicious. My girlfriend (who doesn't usually think like this AT ALL) also pointed out the unusual timing.

Definitely something fishy going on.

Which ? The Data Centre outage or the Bank Holiday ?

[Tumble]

I wouldn't be surprised if the CEO got his normal bonus, i can't remember the last time a CEO stepped down after a publicised major snafu.

Ummm... at the risk of sounding cynical, 3 words.....

David
Cameron
Brexit

Just sayin.....