Microsoft’s preliminary post-incident report says a software bug introduced in July caused the nearly 24-hour Microsoft 365 outage that began November 12, according to BleepingComputer. More than 80% of its networking gateway servers stopped responding, taking down Outlook, Exchange Online, Teams and Copilot.
Three things lined up: the latent bug in a July update to the gateway servers, extra traffic after a Teams failover onto Exchange Online’s network path, and server capacity already cut by an unrelated issue the day before. Microsoft says its load tests missed it because the labs used weaker hardware with fewer cores, and the contention problem gets worse on bigger, newer machines. The Verge adds a real-world cost: a German hotel missed email alerts about high chlorine in its pool during the outage, and 15 people were affected, four of them taken to the hospital.
My take
Two lessons I keep relearning. Test on hardware that looks like production, or you’re testing something else. And never let a safety alert ride on one path. If it matters, send it two ways.
God Bless, and Needs Garlic!