Hidden soft error, heavy load among 3 factors caused Microsoft outage; over 80 pc servers hit

Hidden soft error, heavy load among 3 factors caused Microsoft outage; over 80 pc servers hit

New Delhi, Nov 16 (IANS) The recent major global outage of Microsoft 365 services was caused by a hidden software flaw, an unusually high volume of traffic and a reduction in server capacity, the company said on Monday.

According to these triggers, the US-based software services provider witnessed a global Microsoft 365 outage earlier this month, disrupting services including Teams, OneDrive, SharePoint and Office applications.

In a post on X, the company's Executive Vice President for Copilot, Agents and Platform apologised to customers and shared the early findings of the investigation into the November 12 disruption.

According to Microsoft, the problem started in a networking system that sits in front of many Microsoft 365 services.

A hidden software flaw caused servers to get overloaded and stop responding under certain conditions.

The company said the code change linked to the flaw was first rolled out in July and had been running across the world for more than three months without causing any trouble.

The flaw went unnoticed because the company's testing systems used different traffic patterns and hardware than the real-world systems.

Microsoft said traffic on the affected system rose sharply after Teams traffic was moved from a backup system to the main system about 15 hours before the outage.

At the same time, nearly 14 per cent of server capacity was unavailable because of an unrelated issue from the previous day.

While either factor alone would not have caused an outage, together they exposed the hidden flaw, causing requests to pile up until servers stopped responding during peak traffic hours in the US, the company said.

Microsoft said its automatic systems first tried to send users to healthy servers, but the extra traffic increased the chances of the same failure happening elsewhere.

This set off a chain reaction that spread quickly across the system.

At the worst point of the incident, more than 80 per cent of the affected servers stopped responding, according to the top executive.

The software giant also said a separate issue stopped its internal safety systems, which are meant to contain failures within a region, from working, allowing the problem to spread globally.

Microsoft said recovery was difficult because servers that came back online were immediately hit by pending traffic and stopped responding again.

Its existing automatic recovery systems were not built to restore services after such a large-scale failure.

Engineers eventually found the faulty code and developed an update that restored normal functioning.

Microsoft also said it will release a full review with more details and corrective steps later this week.

--IANS

ag/