soam's home

home mail us syndication

Archive for March, 2010

Netflix + AWS

Recently, I discovered Practical Cloud Computing, a blog run by Siddharth Anand, an architect in Netflix’s cloud infrastructure group. In a recent post, he writes:

I was recently tasked with fork-lifting ~1 billion rows from Oracle into SimpleDB. I completed this forklift in November 2009 after many attempts. To make this as efficient as possible, I worked closely with Amazon’s SimpleDB folks to troubleshoot performance problems and create new APIs.

Why would they need something like this? From another entry titled, Introducing the Oracle-SimpleDB Hybrid, Siddharth writes:

My company would like to migrate its systems to the cloud. As this will take several months, the engineering team needs to support data access in both the cloud and its data center in the interim. Also, the RDBMS system might be maintained until some functionality (e.g. Backup-Restore) is created in SimpleDB.

To this aim, for the past 9 months, I have been building an eventually-consistent, multi-master data store. This system is comprised of an Oracle replica and several SimpleDB replicas.

In other words, Netflix is planning to move many of its constituent services into the AWS cloud starting with their main data repository. This sounded like a pilot project, albeit a massive one, and understandably so given the size of Netflix. If this went smoothly the immediate upside would be Netflix not spending a fortune on Oracle licenses and maintenance. In addition, AWS would have proved itself to be able handle Netflix’s scale requirements.

Evidently things went well as I came across a slide deck detailing Netflix’s cloud usages further:

Fascinating stuff. From the deck, it appears that in addition to using SimpleDB for data storage, Netflix is using many AWS components in for its online streaming setup. Specifically:

  • ec2 for encoding
  • S3 for storing source and encoded files
  • SQS for application communication

I also saw references to EBS (Elastic Block Storage), ELB (Elastic Load Balancing) and EMR (Elastic Map Reduce).

I think for the longest time, AWS and other services of its ilk, were viewed as resources used by startups (such as ourselves) in an effort to ramp up to scale quickly so as to go toe to toe with the big guys. It’s interesting to see the big guys get in on the act themselves.

Replacing my MacBook Pro Drive

Back in grad school, my thesis advisor Brian Smith, to his eternal credit, really put the systems into computer systems where our research was concerned. He also placed the same emphasis on our group and how we dealt with our own computers. We joked that much like Marine Boot Camp, our group members needed to know how to take apart and put together an entire computer in less than minute in order to be able to graduate!

I tried carrying on this entire DIY ethic in my post graduate career. There were stumbles though when I first started dealing with laptops. In 2000, I permanently crippled my Dell Inspiron while taking it apart in order to replace the internal hard drive. It never worked quite as well post operation. It literally was held together by liberal applications of masking tape and glue. I still take pride in being able to sell it to a fly by night computer repair operator in Kolkata sometime around 2005. It probably exists in some incarnation somewhere, fueling some kid’s IIT aspirations right about now.

Given my last experience, I was somewhat nervous about replacing my MacBook Pro’s hard drive. No question it was due – two years of hard labor had squeezed the existing Fujitsu down to cacophonous joint on joint grinding. Each day, I thought, would be its last. However, it wasn’t a straighforward processs. Apple provides a How-To on its site if you want upgrade your MacBook but for your Pro, no dice.

Luckily, help exists online, in particular, here, here, here and here. The basic procedure is the same: you first buy a 2.5″ SATA drive, ideally 7200 RPM for faster performance even though it’s a bigger drain on the battery. I’ve never gone wrong with Western Digital, so I bought a WD 320G Scorpio. Next, you’ll need a 2.5″ enclosure – make sure it can do SATA drives. I learned the hard way. Finally, you’ll need a Phillips and a T-6 Torx screwdriver. I bought all except the Phillips screwdriver (I already own a set) from Fry’s. Not the cheapest but at least they’ll take returns in the first 30 days.

After fitting the WD drive to the enclosure, I hooked it up to the Pro’s USB drive and used SuperDuper to completely clone my main HD into an externally bootable drive. I rebooted the Pro from the external drive to confirm (hold down the option key when rebooting your Pro and, if multiple bootable devices are available, it’ll ask you to choose).

For the actual physical work, I printed out ifixit’s guide and followed it step by step. You have to take out a lot of screws and some parts. To keep track, I placed all the pages in the guide side by side and placed each set of screws next to the pictures, once I completed the step. This was particularly useful when putting everything back together again. My Phillips screwdriver is magnetized, so it holds the screw to itself. This was invaluable as many of the screws in the Pro’s casing are tiny and placing them can be tricky.

It was quite a relief when I put everything back together again, powered up my laptop and, after a brief, yet agonizing period, the Apple logo came on. And soon after, the machine booted up quite happily and my desktop appeared. Now, I have a souped up box and quite possibly have saved my company, Delve, a fair chunk of change in terms of not having to replace my laptop. One of the keys in the keyboard is still loose but I am hoping the tape and chewing gum approach will work in holding it in place!