Company underbosses outsource key infrastructure to outsource reponsibility w/inevitable worldwide failure as result

Sometimes outsourced computer infrastructure fails but then at lest the companies can blame 3'rd parties and say they can do nothing about it. /As if that is a positive.

The truth is that it is still their responsibility and top bosses shouldn't accept such feeble excuses from their management team. The different departments made thier choice and when it fails, even if its a third party fail, they need to still own it. when you outource the running and key parts of your infrastructure to sample Microsoft Azure it will still be you customers problem when it fails, and therefore yours. 

Large serverfarms are often remote from key personell. And the more of it there is the less personell there are per server. Do you trust them to test upgrades thoroughly before mass rollout. Do you thrust them to only mass rollout to a part of the network so that those can be isolated and other parts still function if mistakes are done. Did you check that this is the case, always.

Are you sure your setup are really multisite for redundancy or just physically multisite but on 2 or more closely connected sites that actually all get upgrades at the same time.

The worldwide affected and multiple types of service like airlines, airports, banks, retailers and pharmacies that was affected today by the Crowdstrike update points to a cloud service. And since it was Microsoft servers you can be pretty sure it was the sky service Microsoft Azure that was hit. And that Microsoft did a rapid worldwide rollout without a time between sites to check that all was till running smoothly. A shortcut probably implemented by some cost saving individual/team/boss without enough experience beetween them to realise that sometimes things goes south and that there need to be procedures and safeguards to catch that.


Some critical parts of the infrastructure should still be under direct control by the inidividual companies. At least they can then more diriectly control what level of redundancy and risk they are willing to have/take. 

CEO's need to stop letting their underbosses outsource their responsibilities to non-inhouse setup. Remind them that they can outsource some of the work but never the responsiblity. 

Comments

Post a Comment

Popular posts from this blog

Is this the endgame for Norse, not necessarily

Norse have again announced that they need another capital injection. This time it might be different because last time they needed money the ceo and large owner Larsen said that it was the last time he injected more from own funds. Thi reminds me of the endgame ein Flyr where major investor Braathen said something similar in the autumn. When they agin needed money the following Januar nobody came forward and that was the end for an airline that could have been saved after the October refill with some major mangement changes that could have implemented even stricter cost savings, found income immediately and not paid out cash  immediately for possible charter incomes in the future. If Norse turns to more subleasing out of planes or charters they need to ensure that there will be upfront payments because it will mean upfront costs. Difficuolt concept for underfinanced charterers but rquired for an airline on the brink. Norse management also need to be based closer to where the action...
Read more

When will Boeing's new chief get around to solving problems instead of letting new ones happen

So far the new Boeing CEO is not standing out as a problem solver. His first couple of months has more been a deepening of the Boeing rut. Maybe his immediate and well announced (partial) relocation to Seattle was not such a smart move after all. It certainly have given the machinists the guts to demand back stuff they gave up 10 years ago because Boeing then threatened to move more production away from the area. He needs to be careful with letting his division leutenants run the show by playing hardball with the unions at a time when Boeing needs to ramp up production, not let it grind to a halt altogether. They may have more experience from within Boeing and been seen by the previous CEO as leading lights but it is time for a new regime. The previous one created and proeeded with more trouble than it was worth for both customers and the company itself. Boeing might be too critical to the for all practical purposes duopoly in passenger plane making in the western world, but that deepl...
Read more

Questionmarks over how Boeing saved costs when using plug instead of door

 What on the overwing exits stop them physically from being opened by a passenger with poor survival instincts during flight and did Boeing design engineers remove this safety mechanism from instructions for when installing a plug instead of a door. What other safety features did they omit on the grounds of cost saving in the design for using a door plug instead of a real emergency exit.  Normal cabin doors can't be opened during flight by design since they swing forward into the wind. There are few comments around on how this door (when installed) pivots. Is it up like the overwing exits or is it down since with such a small door there are probably no self inflating slide like the normal cabin doors. Or is it left or right. On the 737-200 the emergency exit overwing door came out loose and into the cabin. On the 737-NG-800 they swing up and out of the way. I doubt it just is pushed out and fall down because that could damage it during testing / maintenance. Would the overwing...
Read more