Comments on: Outage post-mortem http://blog.phanfare.com/2008/11/outage-post-mortem/ Phantastic thoughts from Phanfare, the best online photo and video sharing service in the universe. Wed, 11 Jan 2012 20:35:42 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Andrew Erlichson http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-66165 Andrew Erlichson Tue, 25 Nov 2008 14:39:55 +0000 http://blog.phanfare.com/?p=93#comment-66165 We pay for the bandwidth 2x in our mind. Once when it hits our datacenter and once when we move it to Amazon. Our datacenter, like many, is more focused on how many bytes you serve than how many bytes you absorb since the bulk of the consumed bandwidth is outbound. We pay for the bandwidth 2x in our mind. Once when it hits our datacenter and once when we move it to Amazon. Our datacenter, like many, is more focused on how many bytes you serve than how many bytes you absorb since the bulk of the consumed bandwidth is outbound.

]]> By: Brad http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-65592 Brad Fri, 14 Nov 2008 23:24:14 +0000 http://blog.phanfare.com/?p=93#comment-65592 So you take uploads into your datacenter, process them and then send them to Amazon? It sounds like you are paying for upload bandwidth 3 times: customer to you, and you to amazon twice (once to your bandwidth provider and once to Amazon). Or do you only do that when Amazon is offline? So you take uploads into your datacenter, process them and then send them to Amazon? It sounds like you are paying for upload bandwidth 3 times: customer to you, and you to amazon twice (once to your bandwidth provider and once to Amazon). Or do you only do that when Amazon is offline?

]]> By: Andrew Erlichson http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-65590 Andrew Erlichson Fri, 14 Nov 2008 22:52:58 +0000 http://blog.phanfare.com/?p=93#comment-65590 We actually do cache the recent stuff on our own servers so that we can have Amazon be down and still have Phanfare be mostly up. We cache websized photos and that is only 10% of the data. We can actually cache the entire Phanfare data set in web sized renditions at just 10% of the storage cost of storing the originals; less in fact because our cache does not need to be durable (replicated across datacenters and geographies) We actually do cache the recent stuff on our own servers so that we can have Amazon be down and still have Phanfare be mostly up. We cache websized photos and that is only 10% of the data. We can actually cache the entire Phanfare data set in web sized renditions at just 10% of the storage cost of storing the originals; less in fact because our cache does not need to be durable (replicated across datacenters and geographies)

]]> By: Brad http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-65587 Brad Fri, 14 Nov 2008 22:04:47 +0000 http://blog.phanfare.com/?p=93#comment-65587 If you are using Amazon and are serving up critical information then you have to at least have your own cache if not your own complete copy. That can be prohibitively expensive, especially for something like personal photos where durability is far more important than dealing with a few hours of outage once or twice a year. If you are using Amazon and are serving up critical information then you have to at least have your own cache if not your own complete copy. That can be prohibitively expensive, especially for something like personal photos where durability is far more important than dealing with a few hours of outage once or twice a year.

]]> By: ricky http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-65546 ricky Thu, 13 Nov 2008 20:08:18 +0000 http://blog.phanfare.com/?p=93#comment-65546 Comments were being made such as: "and this is why you have to setup a fail-safe", "the s3 service is great but this just proves you can't rely on it", "My business is effectively closed right now because Amazon did something wrong. I'll have to reconsider using the service now.", "Errors happen, but there MUST be a fail-safe way of reporting them." --------------------------------------- ricky <a href="http://www.casualdate.net.au" rel="nofollow">dating</a> Comments were being made such as: “and this is why you have to setup a fail-safe”, “the s3 service is great but this just proves you can’t rely on it”, “My business is effectively closed right now because Amazon did something wrong. I’ll have to reconsider using the service now.”, “Errors happen, but there MUST be a fail-safe way of reporting them.”
—————————————
ricky
dating

]]> By: Andrew Erlichson http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-65542 Andrew Erlichson Thu, 13 Nov 2008 19:03:03 +0000 http://blog.phanfare.com/?p=93#comment-65542 We are most concerned about the durability of the data versus the availability. Our data is stored at Amazon S3, and they provide excellent durability and very good availability. We actually don't need Amazon to be up to serve must of our traffic. In fact, in the big Amazon S3 outage, most Phanfare users were not aware of the issues. We are most concerned about the durability of the data versus the availability. Our data is stored at Amazon S3, and they provide excellent durability and very good availability.

We actually don’t need Amazon to be up to serve must of our traffic. In fact, in the big Amazon S3 outage, most Phanfare users were not aware of the issues.

]]> By: Ken Donoghue http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-65532 Ken Donoghue Thu, 13 Nov 2008 15:52:44 +0000 http://blog.phanfare.com/?p=93#comment-65532 Customer satisfaction and services availability are understandably Andrew's concerns (Kudos, BTW, for explaining the outage.) Brad's input, as a user of Phanfare, is worth considering though because it's users who often define what an acceptable level of availability is for an application or service. The gap between the two is not nearly as wide as it once was (and ususally the service expectations are reversed! Kudos to Brad). A quick web search will yield a number of solutions that are neither complex nor expensive and can deliver up to four nines ... some better and some with virtualization included. This end of the availability spectrum is changing at an incredible pace, for the better. Customer satisfaction and services availability are understandably Andrew’s concerns (Kudos, BTW, for explaining the outage.) Brad’s input, as a user of Phanfare, is worth considering though because it’s users who often define what an acceptable level of availability is for an application or service. The gap between the two is not nearly as wide as it once was (and ususally the service expectations are reversed! Kudos to Brad). A quick web search will yield a number of solutions that are neither complex nor expensive and can deliver up to four nines … some better and some with virtualization included. This end of the availability spectrum is changing at an incredible pace, for the better.

]]> By: Brad http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-65528 Brad Thu, 13 Nov 2008 13:54:20 +0000 http://blog.phanfare.com/?p=93#comment-65528 4 nines is a waste of time and money for a photo site. If you are getting 3 you are knocking it out of the park. As long as you aren't losing data (other than any potential losses of new data during the downtime), 2 nines is plenty for me. Most downtime is scheduled for off-hours anyway. 4 nines is a waste of time and money for a photo site. If you are getting 3 you are knocking it out of the park. As long as you aren’t losing data (other than any potential losses of new data during the downtime), 2 nines is plenty for me. Most downtime is scheduled for off-hours anyway.

]]> By: anon http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-65473 anon Wed, 12 Nov 2008 17:43:54 +0000 http://blog.phanfare.com/?p=93#comment-65473 "strives for" being the key word -- they aren't hitting that. 99.99 is very hard actually. “strives for” being the key word — they aren’t hitting that. 99.99 is very hard actually.

]]> By: Outage post-mortem « Phanfare Health Status Blog http://blog.phanfare.com/2008/11/outage-post-mortem/comment-page-1/#comment-65468 Outage post-mortem « Phanfare Health Status Blog Wed, 12 Nov 2008 16:19:04 +0000 http://blog.phanfare.com/?p=93#comment-65468 [...] Please see our blog for an explanation of yesterday’s [...] [...] Please see our blog for an explanation of yesterday’s [...]

]]>