|
We have a Lighttpd/Perl/MySQL web service we run on an Ubuntu VPS, and want to add redundacy so that if our datacenter has issues, we stay up. Interested in thoughts and comments on our proposed solution:
Interested to know if anyone has tried doing anything similar, or for any thoughts, comments or suggestions on the above strategy. |
|
Possibly not the answer you're looking for, but have you done a cost benefit analysis to ascertain the true impact of an outage and then compared that with the costs of the kind of redundancy you're talking about? Have you taken into account the added complexity the above setup adds and ensured it won't cause more downtime then the issue you're trying to avoid? True real-time live geographically redundant infrastructure can be extremely complex and in many cases is overkill. A good backup schedule and properly planned disaster recovery plan can often mitigate a datacenter outage (by failing over to your backup datacenter) to just a couple of minutes, while avoiding a lot of the complexity. FWIW, the fact that you're currently running on a VPS was the first indication that what you're looking to do is almost certainly overkill, but I could be wrong. --jeremy |
|
As a followup to some of these "why bother" / "paranoia" comments: We have experienced four outages in the last month within our current datacenter (naming no names, but they are a huge organization in Dallas) which is why we're looking to add redundancy. Each time our services go down we lose a significant amount of money as our advertising partners (Google, etc) are bringing visitors for which we pay, but who can't see our site to convert to sales. While some advertising partners such as Adwords can be "paused" in a short space of time during an outage, not all of them can. Working with small margins at high volume as we are, any extended outage can be very expensive. Although the datacenter claims 100% uptime, things can still happen, as we saw last month. The amount we lost due to the outages so far last month, was far LESS than the cost of a redundant setup in another facility would have been. -J 1
FWIW, my response was definitely not meant to be of the "why bother" OR "paranoia" variety. It's just that you need to have a good understanding of exactly how much downtime actually costs you vs. how much a properly setup redundant setup that won't cause more downtime than it avoids (and I see a ton of these in the real world) will cost. You should also keep in mind how it will impact scaling, administrative tasks, upgrades, etc. Does this wrong can have quite a few long term negative consequences.
(11 Jun '10, 14:22)
jeremy ♦♦
|
|
I would recommend looking into Linux-HA, and heartbeat and pacemaker. I use them for high availability failovers between servers. |
|
Paranoia => greater cost than benefit obtained. Sounds like a Fed job. |
looks well-thought, just not sure about the 4 seconds failover given that (as I understand) they are on two different physical locations. also, have you considered the (odd) case when only the DB fails but the web keeps serving the pages? is this handled properly?
@pmarini in the instance that the DB fails this could get caught by the "monitoring" process, or, I suppose by the CGI scripts which could determine an invalid sql response and take their own IP address offline
The 4 seconds is based on Page 2 Table 1 of http://crypto.stanford.edu/dns/dns-rebinding.pdf - though this is talking about DNS rebinding attacks, the same DNS "Pinning" scenarios in the browser is what we'd be looking at here with dual A records where one A record's IP goes offline and the browser is forced to re-pin the DNS name to a different IP... in theory :)
Propably this question will receive more answers on Serverfault.com. I think, it's very hard question...
Please accept an answer so the question/answer can be finished. Or provide more details so we can help.