cross-posted from: https://feddit.online/c/[email protected]/p/2115831/microsoft-365-outage-leads-to-chlorine-gas-poisoning-of-15-hotel-guests
No mails arrived due to the disruption […]. So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.
Kinda scary to rely solely on email for such alarms.
Yeah, I hate Microsoft as much as the next person, but this is 100% on Phantasialand for trying to cut corners with excessive automation.
It’s not excessive automation – it’s a lack of redundancy for critical safety features.
Anything safety-critical should never have a single point of failure if you can help it. And there was definitely an opportunity here to have some kind of secondary backup alarm system.
I think that automation is fine, but when you start talking about life-critical systems, the bar on failure modes and reliability becomes a lot higher. Like, if you have a sensor for excessive chlorine levels, the default probably should be not to keep pumping more of the chemical in, but to have some “hard” upper level where no more chlorine goes in (and if the two systems aren’t directly-integrated, to at least let the maintenance safety system power off the chlorine system), and for a secondary alarm to be triggered (like, start playing a loud audible alarm) if maintenance doesn’t respond to the email notification and indicate that they’ve dismissed the alert within a given amount of time. That’s disruptive and undesirable compared to just emailing maintenance, but as long as there aren’t any failures, you shouldn’t ever see it. But if you do have it come up, you don’t want the failure mode to be chemical-keep-being-dumped-into-the-system-and-people-being-unaware-of-it.
You want fail-safe behavior.
Automation can increase safety by decreasing time until an issue is addressed, for example. Automation isn’t intrinsically incompatible with safety. Aircraft computers today use TCAS to automatically instruct pilots what to do if they are about to collide (and in fact, today the automated system is explicitly given higher priority than human air traffic controllers — if the two give pilots conflicting directives, the human is the one to be disregarded). But you gotta make your automated procedures have high reliability and handle failure the way you want when the consequences of a failure are high (and that goes for non-automated procedures too).
Excessive automation? That actually seems like a great thing to have automated… just not by email. Nothing stopping them from having a sub ghz radio that sets off a physical alarm somewhere. Improper automation I think would be more apt. A purely local automated alarm sounds great, but the internet shouldn’t be involved imo, at least not on the critical path to remediation.
I wonder if they could have done something like [puts on big brain hat] have an actaul fucking alarm the pool staff and guests can hear.
Well, at least the automation was not artificially intelligent.
I for one think that IT team should be kicked to the curb, but they will probably get raises to figure out that they can use email, healthchecks.io, and ntfy all at the same time for redundancy or they will say we need money to switch providers and set up a gmail account to do the same thing and call that redundant cause you know getting two emails saying the same thing is not going to end up just being ignored.
I used to have IT monitors that would email but also text me. Most folks don’t know they can send an email to (phone_number)@(provider).com and that shows up as a text.
I think major American carriers stopped allowing that, actually
That’s actually terrifying a lot of people don’t even care on checking their mails on a daily basis
Can’t put any blame on MS here. Using email for safety warnings with no redundancy is negligent.
One can put some blame on them, legally speaking.
- First, MS was outside their SLA. That’s right in their service agreement.
- Second, they were so far outside their SLA that it gets into reasonable expectation.
- Third, there will be litigants who assert negligence.
Which of those claims will stick remains to be seen, but “no blame” is incorrect.
This is why I always use Yahoo mail for my critical life safety notifications.
Make sure to throw the system status dashboard up on GeoCities
We were all exposed to this problem: on Friday there was an outage that crippled the US software company Microsoft, including the email program Outlook.
Phantasialand was affected as well. Because of the disruption, no emails arrived, the company told WDR on Saturday. So nobody noticed that the maintenance safety system for the sauna area had long since raised the alarm by email. The automatic dosing system had pumped too much chlorine into the pool.
Email is not a guaranteed delivery system. Now they know.
Wow. Why wire up a blinking lite and speaker in the managers office to the system when you can just rely on fucking email. Pretty sure whoever made it possible to use email intended it to be a backup solution in case no one in charge was on site when the alarm went off.
