Something Wrong with Facebook

Something Wrong With Facebook - Early today Facebook was down or inaccessible for many of you for approximately 2.5 hrs. This is the worst failure we have actually had in over 4 years, as well as we wanted to first off apologize for it. We likewise wished to offer much more technical information on what took place and also share one big lesson discovered.

Something Wrong With Facebook

The key problem that created this interruption to be so severe was an unfortunate handling of a mistake condition. A computerized system for confirming arrangement values ended up creating much more damage than it dealt with.

The intent of the automatic system is to check for arrangement worths that are invalid in the cache as well as replace them with upgraded values from the relentless shop. This works well for a short-term trouble with the cache, however it doesn't function when the relentless store is invalid.

Today we made a modification to the persistent duplicate of an arrangement worth that was interpreted as void. This meant that each and every single customer saw the invalid value and attempted to fix it. Due to the fact that the solution entails making a query to a collection of data sources, that collection was swiftly overwhelmed by numerous thousands of queries a second.

To make issues worse, every time a customer got a mistake trying to query one of the databases it analyzed it as an invalid value, as well as erased the equivalent cache secret. This implied that even after the initial issue had actually been repaired, the stream of questions proceeded. As long as the data sources stopped working to service a few of the requests, they were triggering a lot more requests to themselves. We had entered a feedback loop that didn't permit the databases to recoup.

The way to stop the responses cycle was rather uncomfortable - we had to quit all web traffic to this database cluster, which implied shutting off the site. As soon as the databases had actually recuperated and the source had actually been dealt with, we gradually allowed more individuals back onto the site.

This obtained the website back up and running today, and also for now we've switched off the system that tries to fix setup worths. We're discovering brand-new styles for this setup system adhering to design patterns of various other systems at Facebook that deal even more gracefully with feedback loops and short-term spikes.

We apologize once more for the site failure, and we desire you to understand that we take the performance and also dependability of Facebook really seriously.

Tweet