We’re getting our fingers into a few diverse but complimentary pies, and wanted to let you know about some of the other things we offer that you may not know about, and some new stuff that you definitely don’t know about!

http://www.drak.net/additional.html

Backup MX Services

Backup MX Records help a lot in making sure that you don’t lose mail – if you do a Google search for Backup MX Services you’ll find them offered from a number of places. What you find is them offered for as little as $1 per domain per month, and that’s what you’d pay us if you wanted it.

We only offer it at this price to DrakNet clients. We’ve offered it for a while and we have a number of you using it, but it’s possible some more of you may benefit from it. Backup MX gives you a secondary mail delivery point in case your mail ever goes down and won’t accept anything. It’ll stay there in a holding bin, so to speak, checking periodically to see if your mail server is up. Once it is, it will deliver the mail to you.

What that, you say? We have phenomenal uptime and you would never need that kind of insurance? Well, thanks for saying that and you’d be right if that’s all you had to depend on (and look, things happen – you buy insurance for things that are unlikely but bad). If you install a mailbox, leave a bunch of mail in it and your mail quota fills up, guess what happens to that mail? That’s right. It bounces like a rubber ball on asphalt.

What’s that, you say? You would never do that? Buddy, I got tickets going back years saying that you’re wrong. It is, frankly, the most common reason mail goes bouncy.

http://monitoring.drak.net/

DrakNet Monitoring will allow you to monitor your web site or server routinely, alert you if there is a problem, or alert us if there is a problem – everything is defined by you. This can be added to your DrakNet account or purchased separately via PayPal on the Monitoring Site itself.

We have a 14 day free trial available here, as well as a demo site available if you follow the link above. This is a great buy if you want to monitor more than Apache on your server or VPS, or program it to harrass us for you while you sleep on the off chance anything goes down. If you’re a reseller, you can get a pretty graph to put on your site as well.

DrakNet Monitoring is powered by Hyperspin, so no, you’re not actually being monitored by the same network you’re on. That would just be silly.

DNS Failover and System Monitoring

If your website/service is down you can now automatically send all of your users to another system. This is meant for primarily static sites that need high availability, and is not meant for dynamic sites that are backended by constantly changing information without added configuration.

We can provide multiple sites on separate servers, multiple sites in separate data centers, and many other various configurations. Due to the possibility of several different configurations, pricing will vary on this service (including the hosting accounts for the multiple sites). Please contact us for what you are trying to accomplish, and we’ll get you a quote.

Shared Hosting Data Center Has Shiny New Pipes

Since we’re announcing cool stuff, here’s something you get for free – we’d like to announce since the last “Pipe Crash” connectivity slowdown at Liquid Web that they have brought two new pipes online and they are lit up and moving.

This now puts the Michigan NOC’s (the shared hosting data center) connectivity at twice available what peak time max’s out at. In plain English, a pipe can go down and now there is the capacity to re-route traffic without any routes seeing the latency that was visible the last several times there was a problem.

The Pennsylvania NOC (our VPS and Dedicated Data Center) has also recently upgraded its connectivity as well, and folks on those servers should be seeing faster speeds at peak time.

We got a call from Travis Stoliker, Director of Marketing for Liquid Web Inc., Thursday morning as as we were promised. I’m just going to copy the bullet points for you that we were sent previous to the call.

• Communication procedures
• Staff notification system – Help clearly communicate issues
• PIMS alert updating immediately
• liquidweb.com/support notification updated immediately
• Development making tool to automate black holing of IP’s; This will remove the need for us to wait for ATT
• Adding significant connectivity to mitigate large DOS attack
• Added a wallboard for the networking team to monitor the network status in real time persistently
• The Sonar Monitoring team, which monitors our customers servers, now monitors our network too
• Improved supervisors ability to deal w/ defiant customers which are affecting our network

Our questions previous to this conversation were these: (a) Will there be additional connectivity added so that if one pipe goes down, the others can pick up the slack? (b) Will there be a method in place so that we can get immediate and specific information on the situation and not be told by someone picking up the phone that they don’t know? (c) AT&T. Just AT&T – fix them or drop them. (d) I can see not terminating someone the first time, but the second time, third time? Since this one site was attracting the attack, why was it allowed to remain after it was clear that the procedures in place were not dealing with the situation adequately?

While I would have liked to see these things addressed last month, LiquidWeb did not let the grass grow underneath their feet after it became fairly clear that this was a serious issue that could, to borrow a tech support phrase, be “duplicated” with the same “error”. Repeatedly.

Their response has gone above and beyond “adequate”. They have already addressed our ability to get information, their ability to throw people off who put the network at risk repeatedly by their deliberate irresponsible choices (and while I won’t go into it, there’s someone I’d like to slap around a bit, and they don’t work for LiquidWeb).

Probably the most important developments are that LW is working with AT&T so that they can nullroute themselves, and while that will take some time, the additional connectivity will be here this week to enable re-routing, which was not possible during these last issues.

In short, we’re impressed. Someone once told me that you really can’t judge a company by what they do when things are going well, you judge a company by how they handle things when things are going badly. Frankly, LW didn’t do the greatest job during the last few events, but they clearly have recognized, taken responsibility for, and apologized for what has happened and are sinking a lot of money and time and effort both in making sure it will never happen again, and explaining to people what they are doing to ensure that it will never happen again. They’re not trying to weasel out of anything. That’s important.

So, we’re not moving, everyone breathe, and at least this spurred us to diversify to prepare for things like this a little bit. (And we are, admittedly, getting a kick out of having Weedle all to ourselves). We’re currently eying the Harpertown, but we’re afraid if we get a new dual processor quad core, all the other dual processor dual cores will be jealous.

We’ll let you know.

So, we wanted to update you on what we’ve done – we’ve moved drak.net to its own box at WiredTree (which is, ironically enough, a company founded by some admins that left LiquidWeb to form their own company and who we’ve worked with before). The main DNS server is also shifting to that box – we’ve left both sites and both DNS servers up and will continue to do so for a week to minimize any issues for folks.

This gets us prepared for catastrophe in a number of ways.

First, DrakNet is offsite, so it will remain up during any outage – we now have four points of redundancy. The Billing/Support server is in one place, networkstatus.drak.net is in another, the drak.net site is in yet another, and then the hosted sites are at yet another. All four networks going down at once is highly unlikely, so in almost any catastrophic event, you’ll be able to get a hold of us in or at one of those three places. In addition, we’ve noticed that despite seeing sites down and not being able to see our site, ya’ll still send an email (which doesn’t seem to occur to some of you as a TAD bit pointless), and that solves that issue handily. Should the worst ever happen and we ever have to move out, you’ll still be able to communicate with us about it as if life were normal, and we can get information out to you.

This is not something we anticipate happening – however, it doesn’t hurt to be overly prepared.

We’ve also moved the DNS servers completely out of the primary data center – this enables us to have access to DNS no matter what is going on, and speed up any moves that may need to take place. Again, not something we anticipate needing but it doesn’t hurt to be prepared.

Today, we experienced yet another group of connectivity events that caused downtime episodes off and on all day. Only a group of you were affected (probably about 25% of you), and we’ll explain how we arrived at that in a moment.

In 2005, DrakNet moved to LiquidWeb from Alabanza after a catastrophic database loss which was directly attributable to a long term failure to back anything on some servers up, at all. We’ve now had our servers at LiquidWeb for over three years, and we’ve been with them since they were an aspiring and growing wanna-be big player to the fairly large data center that they are now.

Back then, the system administrators answered the phone – well, they don’t anymore but we still know a lot of them by first name. We don’t NOC hop, and we like to settle in, make a home, and be friendly – in other words, we want a data center that treats us the way we try and treat our clients. We did, and we’ve been phenomenally happy with LiquidWeb.

As a smaller host, our reputation rests squarely upon the shoulders of whoever we do business with, no more so than LiquidWeb since they are the very foundation that we choose to stand upon. We’re not a small web host that chooses to “act big” as if we’re some corporate conglomerate – we lease our servers, we control almost every aspect of them, and we depend on LiquidWeb to do their part for our little boxes in the corner of the NOC. We also choose to be totally honest and upfront about that. By leasing servers, we have quality that we could never achieve at our size, on our own. You benefit from that, and we benefit from that, and the fact is the vast majority of hosting companies do exactly the same thing we do. The Internet is quite a miraculous, boundary dropping thing, and we really believe in this model.

The past few months have been trying – there have been repeated (4 now) periods of semi-downtime due to DDOS attacks. LiquidWeb’s reputation has clearly taken a hit – and that means for those of you affected by these outages, so has ours. There’s no kitschy joke we can interject here – it’s stressful, it’s difficult, it interrupts businesses and activities and communications, and since our business is entirely dependent on these servers purring like kittens, we are no less frustrated, aggravated, and angry as you are when things like this happen. It’s especially painful when you try and do a good job, and the vendor above you is causing you to lose your reputation even though there is nothing at the moment that you can do.

We first want to remind everyone that we went through the migration to the cPanel platform so that we could pick up and get the heck off of anywhere within a day or two with little noticeable interruption in service for anyone, and we are set up to do that. We have full offsite backups at another data center, and we have obtained a second backup server at yet another data center today for double-redundancy. We are moving our main DNS to that tertiary data center in the unlikely event of a catastrophic meltdown, just in case. Today’s events have definitely spurred us to be overly prepared. So, if things went kablooey, neither we or your sites are stuck permanently anywhere.

Now, as to the events of today; there was one site on one client’s server that made someone mad, and got attacked. Repeatedly. Despite LiquidWeb null-routing the attacks, AT&T has to stop them from saturating the connection itself, and it seems just from data we’ve looked at from traceroutes we took ourselves that their responses to these events were anything but acceptable on the part of AT&T. Just as we have to call LiquidWeb for an issue outside our direct control (if we can’t get to the command line, we depend on others to get to the console), so too do they have to call the people that hold their connections for them to do what they need to do, and apparently, this was not a high priority at AT&T (and yes, I am assuming – if it was, in my opinion, it would have been resolved faster). AT&T is one of four pipes in to the data center, and if you drew the short straw you were in the 1 out of 4 that were stuck.

After the last situation, we went up the chain – we are aware that LiquidWeb is ordering more connectivity and hardware to deal with the issue, and we’ll have more specifics for anyone that wants them after Thursday, or nudge us to post it here and we will. Read at your own peril, your eyes might glaze over, though we’ll translate it into English for you as best we can.

We are not prepared to jump ship yet – we have been assured the site being attacked has been terminated off the Network so the risk of further issues has gone down some. We do have questions about why it was allowed to stay when it was such risk to 1/4th of the Network and that hasn’t been adequately answered yet, but we’re still asking. We’re also aware that nuking one site is akin to putting a bandaid on an issue and we have further questions regarding capacity because this certainly isn’t the last time its going to happen. We want to know that what they are setting up will allow for re-routing. We do know meetings are still going on and hopefully we’ll have those answers.

We do want to assure everyone we’re not going to stay if we feel that this will indefinitely continue, but we’re not prepared to nuke a 3 year relationship with what has otherwise been a stellar company to work with because their vendor decided to fall down on the job. We’re preparing to work around our vendor’s potential shortfalls, and we hope that LiquidWeb is taking the time to do the same. Within the next week or two, we feel sure that we’ll know one way or another and, frankly, we trust LW’s integrity based on what they have done for us in the past – my hope is that their intentions matches their actions, and I think given time to sort it out, they will.

If we find out that’s not the case we want to assure everyone that we’re prepared to do whatever we have to do to maintain dependability for all our clients, not just the 3/4s of you that still think we rock because you haven’t seen any down time in ages and didn’t see any today.

Thanks for your patience today and, as always, if you have any questions or comments, let ‘er rip.