Alakzam Downtime
This afternoon, there was a failure on Alakazam during a routine distillation of Apache. While initially, a setting was changed, this setting was in a separate file that could be included or left out of an Apache restart. Once Apache failed, we deleted the lines that were included, and distilled Apache again. All distillations of Apache came back as successful, however, when restarting Apache, it would fail.
Once we checked the configuration file for Apache to see if we could spot the issue, we did – the configuration file being generated was completely and totally empty. We attempted to rebuild it and though it again stated it was successful, the configuration file remained blank. We completely recompiled Apache with cPanel, and Apache was unable to complete a recompile. We then forced an update, recompiled, and it failed yet again. That took about an hour.
What happens now demonstrates one of the reasons that we’re so incredibly open about what type of web host we are.
We are a small (or mid-sized, depending on who’s metrics you’re utilizing to pigeonhole yourself into a meaningless category) web hosting company, deliberately. We don’t really advertise, we just do what we do, people tell people, and that’s how we’ve grown. We are not a web hosting company with an owner that sits around and collects checks while outsourcing every function because its a money-making endeavor, this is a hosting company because I truly love web hosting and I created myself a job that I would love doing.
Very early on, I made the decision to go with Managed Servers despite the extra cost, and I have never wavered from that decision. Yes, it costs us way more for the servers than a lot of our competitors, and yes, I could be making a buttload more money choosing cheapo servers at a crappy data center, or unmanaged servers where they’re just left to spin on their own. Some people have questioned why I “overpay” for managed servers when I know how to administrate a server just fine, and rarely ever have to go “up tier” for anything at all.
This is why. Because there are times when 3 heads on duty can’t solve a problem, and there are times when you can’t deliver a Class A experience unless you’re a Class A bohemoth with staff to spare and experience with hundreds of machines and thousands (ok, millions) of problems. Tonight we at DrakNet hit a frustrating wall in our attempts to solve the issue.
Luckily, we are in a stellar data center, with an absolutely incredible staff that we can call on at any time for any reason and we can borrow them to come work for us for a bit. And that’s why you don’t choose “cheap” – you can have it fast, right, or cheap, just pick any two. (Although, ironically, I pay less at our current data center than I did at another data center that sucked eggs.) We generally don’t call on them for a whole lot because we’re pretty self-sufficient, but when we call on them we need them OMG RIGHT NAO!!
And we get them.
So, thanks go out to Scott Sullivan and Patrick Hawkins at Liquid Web, who were able to wrestle Apache into submission. We’d love to tell you why it happened, but no one really is totally sure why it happened. The credit, though, goes to them because the only thing we really did was yell help, and they kicked her back up. 
A lot of hosting companies hide their dependence on their data center, or the roles that their data centers play in their success. Our set up as a host is not much different than some of the largest players on the market who don’t have their own data centers – no, I won’t name names. You know who you are.
We’ve never done that because we don’t think its a problem that we’re not almighty eggheads that can solve every single thing all the time, and we feel intensely about the role that Liquid Web plays in our success. We also feel its our responsibility to be honest about our company, and there’s no reason to hide or minimize our business set up, or play down their role to make ourselves seem bigger or more than we are. We feel its a great asset to you have two companies simultaneously watching over your sites who are truly intent on making sure everything runs smoothly, and Liquid Web deserves the credit for giving us a fabulous foundation to serve you.
Their safety net’s not too shabby, either. 
If you wonder why we chose to discontinue offering dedicated servers and VPS’s and partner with Liquid Web, why we recommend you to their server service when there is far more profit to be made selling slices and boxes, its because we truly think they’re the best and we couldn’t begin to compete in an arena that they accomplish so much in. We’d be remiss in trying to sell an inferior product. (Yes, we think we kick their ass in shared hosting, but that’s another post for another time).
So, thanks Liquid Web for saving our rear yet again, and thanks clients for being patient and not screaming at us while we fought what turned out to be a serious issue.
Though if you need to, there’s a comments section – have at it!
