On Monday 4 October 2021 Facebook, Instagram and WhatsApp suddenly went down. Kate Bevan explains what happened to the tech giant and what we can learn.
What were you doing during the Great Facebook Outage of 2021? While it wasn’t quite up there with some of history’s defining talking points, the absence of Facebook and its sister services Instagram and WhatsApp was a stark reminder not only of how connected we all are, but how much we rely on those connections.
And it’s also a wake-up call about the dangers of putting too many eggs into a single basket – both for us as individuals, but also for businesses that rely on these platforms, and for Facebook itself.
What happened to Facebook?
It seems that something went wrong during a configuration tweak at Facebook. It’s important to note that this almost certainly wasn’t a cyber-attack on Facebook. We don’t know – and may never know – if it was a genuine error, or something internal, but I am going with the principle of Hanlon’s Razor, which states ‘never attribute to malice that which is adequately explained by stupidity’.
The effect of this error meant that Facebook stopped telling the rest of the internet that it exists. It turns out that Facebook doesn’t outsource any of its networking to third-party cloud providers like Amazon: it does everything itself.
Because Facebook handles all its networking this way, once it had vanished from the internet, none of its internal systems worked. That meant engineers couldn’t remote in to fix the error, and when employees began showing up at Facebook offices, their entry cards didn’t work and they couldn’t get in to the buildings. Facebook’s internal email and messaging systems weren’t working, either.
The nerdy bit
You can skip this bit if you’re not inclined to be geeky, but for those who are, this was a BGP routing failure. BGP stands for ‘border gateway protocol’ – it’s the system by which big standalone networks like Facebook connect to the rest of the internet via big routers. Those big routers hold maps of the routes that packets of information can use to make their way around the internet.
Without those maps – which are constantly updated – none of the standalone, or ‘autonomous networks’, can communicate with each other.
For some reason, what should have been a routine update from Facebook to those maps went wrong. Because it couldn’t announce its presence to the internet, Facebook in effect vanished from the internet.
My personal favourite explanation of what happened came via Twitter last night:
TLDR. Facebook closed the hotel door with the keycard and its passport inside. Now it is nude in the foyer trying to convince the night porter that its name is Facebook. https://t.co/Dib9JAp2m0
— The Blindboy Podcast (@Rubberbandits) October 4, 2021
The knock-on effect
Of course, it wasn’t just Facebook that went down: Instagram and WhatsApp, both of which are owned by Facebook, also went dark. I first noticed this when I was chatting with Harry Rose on WhatsApp.
There were two big fallouts: first, an awful lot of people use their Facebook profile to log in to other websites and services, and so couldn’t connect to those.
Using a social platform to sign in to other sites is considered a bad idea by security folk: first, it means you’re sharing all that data of your shopping and browsing with Facebook (or Google, or Twitter if you use those logins). Second, it means that if your Facebook password is stolen or compromised, cyber-criminals then can get in to all those other sites you’ve connected to Facebook.
And third, as we discovered yesterday, if Facebook goes down, you’re locked out of everything else, too.
The second big fallout was that other services started creaking under the strain. Twitter was the most obvious victim of this: as people couldn’t connect on Facebook, they turned to Twitter to find out what was going on. Additionally, people started hitting the refresh button on Facebook/Instagram/WhatsApp, which meant networks were swamped with traffic and other parts of the internet slowed down.
What does it all mean?
The Facebook outage reveals that it’s not the best idea to put all your eggs into one basket. Facebook engineers will be grappling with that lesson today, while the rest of us should think about having other systems in place for our personal comms, and for our work comms too.
I spent a fair bit of the evening on Twitter and Signal discussing the outage, and today I’m thinking about replicating some of my WhatsApp groups on Signal as a back-up option.
What apps do you use to communicate with your friends, family and colleagues? Did you even notice that all these services were down last night? And if so, did you go elsewhere?
Let us know your thoughts on what happened in the comments below.