'By 7:00am PST there were an insufficient number of backend servers to meet our capacity needs,' according to the report, and Slack went down hard across the world. Slack engineers were not alerted to the problems until around 6:45 a.m.

That forced healthy servers to handle an increasing amount of demand as more and more servers were tagged as 'unhealthy' due to their lack of responsiveness, thanks to the networking issues. Over the next hour, packet loss caused by the networking problems led Slack's servers to report an increasing number of errors. Slack declined to comment beyond confirming the authenticity of the report. PST we began to experience packet loss between servers caused by a routing problem between network boundaries on the network of our cloud provider.' A source familiar with the issue confirmed that AWS Transit Gateway did not scale fast enough to accommodate the spike in demand for Slack's service the morning of Jan. The hours-long outage that kicked off the 2021 working year for Slack customers was the result of a cascading series of problems initially caused by network scaling issues at AWS, Protocol has learned.Īccording to a root-cause analysis that Slack distributed to customers last week, 'around 6:00 a.m.

Slack outage