Today, we experienced yet another group of connectivity events that caused downtime episodes off and on all day. Only a group of you were affected (probably about 25% of you), and we’ll explain how we arrived at that in a moment.

In 2005, DrakNet moved to LiquidWeb from Alabanza after a catastrophic database loss which was directly attributable to a long term failure to back anything on some servers up, at all. We’ve now had our servers at LiquidWeb for over three years, and we’ve been with them since they were an aspiring and growing wanna-be big player to the fairly large data center that they are now.

Back then, the system administrators answered the phone - well, they don’t anymore but we still know a lot of them by first name. We don’t NOC hop, and we like to settle in, make a home, and be friendly - in other words, we want a data center that treats us the way we try and treat our clients. We did, and we’ve been phenomenally happy with LiquidWeb.

As a smaller host, our reputation rests squarely upon the shoulders of whoever we do business with, no more so than LiquidWeb since they are the very foundation that we choose to stand upon. We’re not a small web host that chooses to “act big” as if we’re some corporate conglomerate - we lease our servers, we control almost every aspect of them, and we depend on LiquidWeb to do their part for our little boxes in the corner of the NOC. We also choose to be totally honest and upfront about that. By leasing servers, we have quality that we could never achieve at our size, on our own. You benefit from that, and we benefit from that, and the fact is the vast majority of hosting companies do exactly the same thing we do. The Internet is quite a miraculous, boundary dropping thing, and we really believe in this model.

The past few months have been trying - there have been repeated (4 now) periods of semi-downtime due to DDOS attacks. LiquidWeb’s reputation has clearly taken a hit - and that means for those of you affected by these outages, so has ours. There’s no kitschy joke we can interject here - it’s stressful, it’s difficult, it interrupts businesses and activities and communications, and since our business is entirely dependent on these servers purring like kittens, we are no less frustrated, aggravated, and angry as you are when things like this happen. It’s especially painful when you try and do a good job, and the vendor above you is causing you to lose your reputation even though there is nothing at the moment that you can do.

We first want to remind everyone that we went through the migration to the cPanel platform so that we could pick up and get the heck off of anywhere within a day or two with little noticeable interruption in service for anyone, and we are set up to do that. We have full offsite backups at another data center, and we have obtained a second backup server at yet another data center today for double-redundancy. We are moving our main DNS to that tertiary data center in the unlikely event of a catastrophic meltdown, just in case. Today’s events have definitely spurred us to be overly prepared. So, if things went kablooey, neither we or your sites are stuck permanently anywhere.

Now, as to the events of today; there was one site on one client’s server that made someone mad, and got attacked. Repeatedly. Despite LiquidWeb null-routing the attacks, AT&T has to stop them from saturating the connection itself, and it seems just from data we’ve looked at from traceroutes we took ourselves that their responses to these events were anything but acceptable on the part of AT&T. Just as we have to call LiquidWeb for an issue outside our direct control (if we can’t get to the command line, we depend on others to get to the console), so too do they have to call the people that hold their connections for them to do what they need to do, and apparently, this was not a high priority at AT&T (and yes, I am assuming - if it was, in my opinion, it would have been resolved faster). AT&T is one of four pipes in to the data center, and if you drew the short straw you were in the 1 out of 4 that were stuck.

After the last situation, we went up the chain - we are aware that LiquidWeb is ordering more connectivity and hardware to deal with the issue, and we’ll have more specifics for anyone that wants them after Thursday, or nudge us to post it here and we will. Read at your own peril, your eyes might glaze over, though we’ll translate it into English for you as best we can.

We are not prepared to jump ship yet - we have been assured the site being attacked has been terminated off the Network so the risk of further issues has gone down some. We do have questions about why it was allowed to stay when it was such risk to 1/4th of the Network and that hasn’t been adequately answered yet, but we’re still asking. We’re also aware that nuking one site is akin to putting a bandaid on an issue and we have further questions regarding capacity because this certainly isn’t the last time its going to happen. We want to know that what they are setting up will allow for re-routing. We do know meetings are still going on and hopefully we’ll have those answers.

We do want to assure everyone we’re not going to stay if we feel that this will indefinitely continue, but we’re not prepared to nuke a 3 year relationship with what has otherwise been a stellar company to work with because their vendor decided to fall down on the job. We’re preparing to work around our vendor’s potential shortfalls, and we hope that LiquidWeb is taking the time to do the same. Within the next week or two, we feel sure that we’ll know one way or another and, frankly, we trust LW’s integrity based on what they have done for us in the past - my hope is that their intentions matches their actions, and I think given time to sort it out, they will.

If we find out that’s not the case we want to assure everyone that we’re prepared to do whatever we have to do to maintain dependability for all our clients, not just the 3/4s of you that still think we rock because you haven’t seen any down time in ages and didn’t see any today.

Thanks for your patience today and, as always, if you have any questions or comments, let ‘er rip.

Tags: connectivity, downtime, routing

This entry was posted on Tuesday, May 6th, 2008 at 7:00 pm and is filed under DrakNet News. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.