Views from Phanfare CEO and Co-founder Andrew Erlichson

Link Surviving an Amazon S3 outage

We use Amazon S3 to store our 80 terabytes of photos and videos. We like the service and it works well. Yesterday, it went down for nearly 8 hours. And during that time, we were mostly up. Cloud computing is all the rage, but sometimes, the weather is really bad and you can’t see the clouds. We planned for that rainy day. Hence, on a day when Amazon S3 was entirely down, I was at the pool, literally. I will tell you about how we did it.

When users upload photos and videos, we first move them to our own servers. In the background, we send the data to S3. If Amazon S3 goes down, we can buffer data for up to two days before we notice. By buffering, we remove the real time requirements of Amazon S3 being up for our users to upload data. We can’t buffer indefinitely, but we are betting than an Amazon S3 outage longer than 2 days is very rare. We always believed short outages would occur. In fact this, is is not the first one.

For serving photos and videos, we act as our own content distribution network (CDN) and cache the hot data. That means that users can view most recent photos and videos, including what was recently uploaded.

All this caching and buffering is done outside of Amazon. We don’t use Amazon’s compute cloud (EC2) for that. We have considered moving more of our system to Amazon Web Services. It is unfortunate that EC2 was built to require S3 to be up in order for to it run. New instances are loaded from S3. So an S3 outage is correlated with an EC2 outage.

Photo and video sharing services that did not plan for S3 outages were completely down yesterday. We estimate that most of the cost savings for our business comes from outsourcing the storage. While we could save some additional money by using EC2, it is not as dramatic as the S3 savings. Hence, we will have to carefully consider before we put all our eggs in that basket.

  • http://www.bradmurray.com Brad

    I don’t understand why Amazon doesn’t bolt a shell like that onto S3. It seems that anybody using S3 seriously has to do it themselves which makes it just another hurdle to obtaining larger partners. I understand that it is a complicated system (regardless of the name), but the interface should be simple (which it pretty much is) and it should be rock-solid (which it isn’t). I’ve been evaluating this since pre-beta and I still don’t feel that it is worth the cost or risk. I currently store my own stuff much cheaper, but I have the management headache that I’d love to outsource. My biggest concern to this day is that they still charge .15/GB/month which they charged almost two years ago. My costs have been cut in half since then.

  • Ben C

    THIS is why I host my photos and videos on Phanfare rather than that “not-so-smug now are we” competitor of yours! Good job Phanfolks!!

  • Pingback: how to service rv ac

  • Brad

    Ben – Was Smug down because of S3? I know they do a lot of rendering and stuff on EC2 and my understanding is that EC2 was down because of the S3 outage.

  • anand

    hi andrew-
    quick question, i remember your post from a year ago where you explained that s3 was only used for backup/archiving. is s3 now being used as primary storage with phanfare’s DC used for hot storage? (if so, was this a result of amazon’s SLA for s3?) sorry if you already explained this somewhere.

    thanks

  • max191

    Awesome blog. Very informative. Your blog is very valuable from the blogger’s mind. Keep it up!
    regards
    charcoal grill

  • Salmon67o0

    I subscribed to your blog when is the next post

    respect
    poly banger
    ______________________________________________
    cheap wow accounts | washington dui attorney | herbal skin care products

  • http://www.furnitureoutletworld.com/ Furniture in North Carolina

    I digged this for more news from you.

    respect
    FERMINA NEIDA
    ______________________________________________

    Back to Phanfare blog home »

© 2007-8 Phanfare, Inc.