Facebook, apps suffer ‘epic’ outage
Service is slowly coming back after global disruption
SAN FRANCISCO — Facebook and its family of apps, including Instagram and WhatsApp, went down at the same time Monday, taking out a vital communications platform used by more than 3 billion people around the world and adding heat to a company already under intense scrutiny.
Facebook’s apps — which include Facebook, Instagram, WhatsApp, Facebook Messenger and Oculus — began displaying error messages around 11:40 a.m. Eastern time, users reported. Within five minutes, Facebook had disappeared from the internet.
Hours later, the sites were still not functioning in most places, according to Downdetector, which monitors web traffic and site activity. But the service was slowing returning, a Facebook spokesman said.
“This is epic,” said Doug Madory, director of internet analysis for Kentik Inc., a network monitoring and intelligence company.
Technology outages are not uncommon, but to have so many apps go dark from the world’s largest social media company at the same time was unusual.
Facebook’s last significant outage was in 2019, when a technical error affected its sites for 24 hours, in a reminder that even the most powerful internet companies can still be crippled by a slip-up.
This time, the cause of the outage remained unclear.
Several hours into the incident, Facebook’s security experts were still trying to identify the root issue, according to an internal memo.
Two members of its security team, who spoke on the condition of anonymity, said it was unlikely that a cyberattack had taken place because one hack was unlikely to affect so many apps at once.
Computer scientists speculated that a bug introduced by a configuration change in Facebook’s routing management system could be to blame. Colombia University computer scientist Steven Bellovin tweeted that he expected
Facebook would first try an automated recovery in such a case. If that failed, it could be in for “a world of hurt” — because it would need to order manual changes at outside data centers, he said.
“What it boils down to: running a LARGE, even by Internet standards, distributed system is very hard, even for the very best,” Bellovin tweeted.
Andy Stone, a Facebook spokesperson, posted on Twitter: “We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.”
The outage caused outrage and mirth online, as Facebook and Instagram users turned to Twitter to lament and poke fun at their inability to use the apps.
But the outage was a blow to small businesses and others that rely on the platform to conduct outreach and advertising and to millions who use Facebook and its apps to communicate with friends and family across the world.
Inside Facebook, workers scrambled because their internal systems also stopped functioning.
The company’s global security team “was notified of a system outage affecting all Facebook internal systems and tools,” according to an internal memo sent to employees. Those tools included security systems, an internal calendar and scheduling tools, the memo said.
Employees said they had trouble making calls from work-issued cellphones and receiving emails from people outside the company. Facebook’s internal communications platform, Workplace, was also taken out, leaving many unable to do their jobs. Some turned to other platforms to communicate, including LinkedIn and Zoom as well as Discord chat rooms.
Some Facebook employees who had returned to working in the office were also unable to enter buildings and conference rooms because their digital badges stopped working.
Security engineers said they were hampered from assessing the outage because they could not get to server areas.
Facebook’s global security operations center determined the outage was “a HIGH risk to the People, MODERATE risk to Assets and a HIGH risk to the Reputation of Facebook,” the company memo said.
A team of employees was soon dispatched to Facebook’s Santa Clara, California, data center to try a “manual reset” of the company’s servers, according to an internal memo.
Several Facebook workers called the outage the equivalent of a “snow day,” a sentiment that was publicly echoed by Adam Mosseri, the head of Instagram.