soam's home

home mail us syndication

Archive for software engineering

Deployment at Facebook

Facebook at Mozcon - Alex


An Ars Technica article, A behind-the-scenes look at Facebook release engineering, has been making the rounds this morning, yielding some fascinating behind the scenes details about FB’s release process. The whole thing is worth a read but some things stood out for me:

  • the Release Engineering team have their own built in bar. The booze must alleviate the pre-release tension!
  • very frequent deployments. It’s not unusual for code to be released daily.
  • the entire FB codebase is bundled into a single 1.5GB binary and pushed out to FB servers everywhere
  • they use bittorrent to distribute the binary since point to point pushes are that much more inefficient
  • they fully expect a percentage of servers to fail the upgrade. This can be due to hardware or network issues.
  • DevOps i.e. FB developers are on the hook for release and performance. This is part of the “”developers on the wrong end of a pager” approach that companies are increasingly taking. Having toiled as defacto Director of Ops at our startup for quite a while and having worked with Ops as well, I’ve seen all sides of the issue. What FB does here is similar to Amazon.
  • The release team actually rate developers depending on deployment smoothness with subsequent impact on their job performance reviews. Rollback is a last resort.
  • Facebook’s HipHop php compiler reduced CPU requirements by 50%. That’s pretty amazing.

The last mind blowing detail I found:

The many data sources tracked by Facebook’s internal monitoring tools even include¬†tweets about Facebook. That information is displayed in a graph with separate trend lines to show the change in volume of positive and negative remarks.¬†This is useful, since one of the things that people do when they encounter a technical problem on a social network is complain about it on a different social network.