So, when we started our redesign, we made a discovery – while our spiffy Javascript fly out menu was awesome in that it could be used across multiple servers, it was cross-compatible, and the code could load once no matter where you traveled making it pretty fast once you cached it, crawlers couldn’t see the darn thing.

Just couldn’t see it. There were pages Google didn’t have any idea we had, and were never going to even get a shot of getting indexed because we didn’t give Google any way of finding them.

Yeah, ok, big woops.

As you can see from the graphic to the right, what Google can see and what Google can’t see is something we take very seriously, as even with our site a mess and most emphatically not optimized for search engines they still sent us an awful lot of traffic and slapped us with a PR5 (which ain’t too shabby for a little web hosting company).

Our answer was sitemaps.

Creating a Site Map

So, what are sitemaps?

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site. (sitemaps.org)

You don’t need to built them by hand, though – a Google search for “sitemap generator” will find both online and downloadable sitemap generators. We decided to use WonderWebWare Sitemap Generator.

When we first used it, we had around 750 pages on the site – which was astounding considering I didn’t remember writing 750 pages worth of stuff. What was happening was that the WordPress Blog has the same post indexed several times – by category, by keyword, and on and on running the risk of the search engines seeing the posts as duplicate content and nuking them altogether.

To address this, we split the site into two sitemaps, one that we used the SiteMap Generator for (the main site), and one sitemap just for the blog in the form of a WordPress Sitemap plugin (XML Sitemap Generator for WordPress 3.2.2). We placed both of those sitemap locations in our robots.txt file to let the search engines know we had them, like so:

Sitemap: http://www.drak.net/sitemap.xml
Sitemap: http://www.drak.net/news/sitemap.xml

If you visit those links, you’ll see we whittled the site down to 37 links, and the Blog sitemap lists each post only once, with the added benefit that it will regenerate each time a post is published to add its content to the list so we don’t have to manually mess with it.

Once you get a sitemap created, you should go the extra mile and submit your sitemap to the search engines – Google Webmaster Tools allows you to claim your site with Google, submit the locations of your sitemaps, and will give you an overview of any errors it finds when crawling your site. For Yahoo! there’s Site Explorer, and for Bing there’s Webmaster Center.

Google has design advice located here, and we kind of flunked the first set from their perspective due to the Javascript navigation, which was:

Make a site with a clear hierarchy and text links. Every page should be reachable from at least one static text link.

so for good measure, we used WonderWebWare Sitemap Generator to create an HTML site map as well which, depending on your navigation, you may wish to think about doing.

So, what started my obsession with organizing the site? Well, DrakNet has been around nearly 12 years now, and in that time the site has “evolved”.

When I say “evolved”, I mean that I kept changing it and sometimes I didn’t clean out everything I wasn’t using at the end so that underneath, it was a mess – including my “I love tables, long live tables” design.

I knew I should do something about it, and after reading Benny Crampton’s post on WordPress and Caching, and after watching Firebug and Google’s Page Speed rate my blog’s speediness as “OMG, did you really put that thing on the web?” I realized that I needed to roll up my sleeves and get some work done.

We’ve already talked about mod-deflate, which was the easiest thing to implement. Today, we’re going to talk about caching specifically as it relates to WordPress blogs.

Benny writes:

Caching (from wikipedia): “In computer science, a cache (pronounced /?kæ?/ kash) is a component that improves performance by transparently storing data such that future requests for that data can be served faster. The data that is stored within a cache might be values that have been computed earlier or duplicates of original values that are stored elsewhere.”

That’s right! All of the sudden that 150 file and 450 database reads go down to 1: the page that was generated previously and stored in cache. I cannot adequately express how much of a difference caching makes. I have seen servers serving a single site that got Dugg go from repeatedly crashing to serving 7Megs of traffic, just by enabling caching.

While WordPress doesn’t have any built-in caching, a number of 3rd party developers have created caching plugins for WordPress which offer various levels of simplicity and control.

Benny’s cache plugin of choice is WP Super Cache:

http://wordpress.org/extend/plugins/wp-super-cache/

which seems to be pretty popular. While I did get it installed and working with success, it also broke my Disqus comments and I had to hack the mod-security rulesets provided, so I wasn’t overly thrilled with the amount of tweaking. The DrakNet blog uses

http://wordpress.org/extend/plugins/w3-total-cache/

which is used by sites like mashable.com, briansolis.com, pearsonified.com, ilovetypography.com, noupe.com, webdesignerdepot.com, freelanceswitch.com, tutsplus.com, yoast.com, css-tricks.com, css3.info and others.

While much more configurable than WP Super Cache and in some respects somewhat potentially over a newbies head, it comes with a nice set if features as soon as you enable it and can be used out of the box to a WordPress site’s benefit.

If you use both caching and compression, it should dramatically improve your blog visitors experience, especially if your coding isn’t quite as flawlessly perfect as it should be. Caching and compression are no replacement for good code, optimizing images, and creating a good design, but this is one of those cases where it takes good and makes it better and takes bad and makes it a little better than it was.

For more information on why caching is an excellent idea, read Benny’s entire post on caching, which explains the benefits in clear, non-technical terms.

The redesign of DrakNet’s site was partially vanity, and partially SEO. Now, when we say SEO, we do not mean cultivating back links from random sites, jockeying with keywords that have nothing to do with us, or randomly finding blogs to comment on just so we can get a link to our site in their comment area.

Our SEO undertaking was all about organization, and speed. We’re going to take each thing we did piece by piece and share it with you.

Today, we’re going to talk about compressing your site.

Let out some hot air

Mod-deflate is an optional module for the Apache HTTP Server, Apache v2 only. Based on Deflate lossless data compression algorithm that uses a combination of the LZ77 algorithm and Huffman coding. This module provides the DEFLATE output filter that allows output from Apache HTTP server to be compressed before being sent to the client over the network. (Lifted word for word from http://en.wikipedia.org/wiki/Mod_deflate)

In English?

It takes your files, squishes them, hands them over to your visitors squished, and your visitor’s browser (if its a modern browser) unsquishes them so they can view the page. This will save your bandwidth, and your visitor’s loading time making your site faster. It does this by trading off resources, as the act of the server squish takes more of the server’s CPU than normal.

None of you are currently using mod-deflate. You know how I know none of you are using mod-deflate? We just installed it on Friday evening on all servers, and no one’s ever asked for it.

If your site currently rates well with regard to speed, you may not need to worry about this. You are going to trade off CPU usage, and we do have a ceiling on that so you should think carefully about how busy your site is, and whether you think that you can afford the CPU hit on your account. On the other hand, if your site is one that is competing for the race up the Google rank or you have a serious concern about having the absolute best possible experience for your visitors with regard to speed, this is something you may want to implement.

And implementing it is pretty much a snap. While it runs on all servers right now, it won’t compress anything until you tell it to on your individual site, and the way that you give it marching orders is through .htaccess. Here’s ours:

# compress certain file types by extension:
<FilesMatch "\\.(js|css|html|htm|php|xml)$">
SetOutputFilter DEFLATE
</FilesMatch>

#exempt old browsers
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

This will squish our pages, javascript, CSS files, php files, and xml pages, but not any of your graphics. Graphics should be web optimized, anyway, and running them through deflate generally won’t save you enough to justify the CPU cost of doing it. It will also serve older browsers a plain old uncompressed page if any happen to come knocking.

Again, the warning is that this will up the CPU cost of serving the pages while lowering the bandwidth and speeding up the page. How high the CPU cost will be will depend on the size of your pages, and your traffic.

If your site’s implementation causes a problem, we will definitely contact you and let you know. You can also implement it, and email support to ask us to look up your site’s daily average resource usage for the days proceeding your implementation and the days during its usage, and we’ll be happy to tell you so you have an idea.

Search Engine Optimization (SEO), and internet marketing itself, is often divided into two different camps: white and black hat.

Now, before we get started explaining the two, let me point out that the spammers who constantly post garbage on your blogs, exploit the holes you don’t patch, and who try and squeeze a viagra link anywhere you let them are doing black hat SEO – so keep that in mind before we offer some explanations that get you all excited about rocketing up the Google Page Rank charts.

Whitehat SEO, the more ethical of the two forms of SEO, relies on strictly and accurately following the terms of search engines and website directories, largely by generating links organically. Blackhat SEO, on the other hand, is less ethical, significantly less effective as a long-term business strategy, but often valuable for its speed.

To distinguish between the two different forms of SEO, it is worth going back to their roots.

White and black hat are not divisions that were thought up specifically for SEO (or hacking), but ideas that originated in western movies. Drawing inspiration from the John Wayne styled heroes of the past, ethical SEOs branded themselves as “whitehat” after the type of cowboy hat the western hero often wore. In contrast, the blackhat SEO world embraced the villainous style – dark clothing and a distinct black cowboy hat.

Blackhat SEO is lucrative. There is no doubt about it – if it wasn’t making someone money, people wouldn’t do it. From spamdexing to indirect and somewhat stealthy blog commenting, there are hundreds of ultra-effective ways for blackhat SEOs to boost their websites’ online influence and search engine rankings. Blackhat-powered websites typically grow rapidly, gaining hundreds of indexed links within days, and occasionally hundreds of thousands within just weeks.

This is generally carried out through the use of automated software – you didn’t think all those comments were done manually, did you? From forum spamming applications to blog auto-commenter tools, the blackhat world has developed its own ultra-lucrative software industry. Of course, as the actions are generally automated and carried out in bulk, blackhat SEO tends to have little long-term value as search engines can easily pick up on the traces and linking footprints and de-index the offending website.

Whitehat SEO, while having only a fraction of the rapid growth potential that blackhat SEO boasts, is more valuable to online marketers and businesses that are focused on long-term internet goals. While blackhat-powered pages tend to climb through search engine rankings relatively quickly, they are forever at risk of being permanently removed from the search engine results altogether.

While this is not a concern for hit-and-run style online businesses (which doesn’t apply to any of those of you hosting with us, I’m sure), it most certainly is for businesses that are focused on long-term branding and internet marketing (which should be everyone hosting with us).

If your business values long-term clients, customers, and branding efforts, it is clear that whitehat SEO is the right choice to make. Despite the longer climb up the ladder and slower movement through the ranks, it is the only choice that offers long-term security.

Stop emailing me and asking me for a Dedicated IP address because you read that this utterly fantabuolous guru swears that you can pull one over on Google and make them think you have your own server, it will make your site faster, grant you first choice in resources, and in general just immediately rocket you to PR 10.

After I stop laughing, I will tell you no.

Every domain name that has a web site on it or a computer or network at the end of it has an IP address assigned to it. An IP address is the “real” address of the website or server or computer or network. Domain names came along because it wasn’t exactly fun to try and remember long IP numbers like 123.456.789.101. Most of the time now in shared hosting when there are a bunch of sites sharing one server those same sites will share one single IP address.

There’s a reason that’s more popular, and it’s not “laziness” or “ease” on the part of the hosting companies. IP4 addresses are running out, fast. (See IPv4 Address Exhaustion on Wikipedia for more info on it). The American Registry for Internet Numbers is in charge of doling out those precious IP addresses, and now, they make you justify exactly why you need them.

As SEO (Search Engine Optimization, or better known as “trying to pull one over on Google so Google will like you better”) has grown, there are persistent myths flying around that paying for a Dedicated IP on a shared server will somehow get you better rankings on the search engines. A whole host of completely bogus myths have popped up surrounding the mythical dedicated IP address, and while I can’t say for sure whether Google really cares whether you’re on a shared IP or a Dedicated one since what they use for PR rankings is proprietary and they’re pretty tight-lipped about it, I can tell you that some of the justification and reasoning being thrown around by the “experts” who apparently haven’t got a clue about how a web server functions is just bunk.

This article written in 2007 touches on a lot of the more common myths that are still flying around, and so we’re going to use it as a springboard to tell you why these suppositions are horse pucky.

One of the reasons using a dedicated IP can have a positive influence on rankings is because the engines take into consideration how fast your site loads in comparison to other sites. If you’re sharing your IP with 500 other Web sites, the server, like all good customer services departments, will deliver the files in the order they received them. If there are 10 people ahead of your visitors in line, they’re going to have to wait, resulting in a slower page load times and frustrated users.

If there are ten people hitting the server waiting in line to use Apache to view a web site, they’re going to be in line whether you have a dedicated IP or a shared IP. A dedicated IP doesn’t get you your own Apache server or anything like your own dedicated server. It just gets you your own special number on the same server. You still have to wait in line for services that are shared just like everyone else. The server does not load pages based on whether you paid extra to have your own super special IP address; the idea that it does is a myth that seemingly won’t die.

There’s only one way your own dedicated IP address will get your site priority service on the machine – if your IP address sends people to your own machine where you are the only one on it.

Sharing an IP address also doesn’t allow you to control who your neighbors are. If you’re sitting on the same IP as a gambling site, a porn ring, a Viagra dealer, and a priest, and one of those addresses gets banned by the search engines, you’re banned too. Search engine’s don’t just ban domains, they ban whole IP ranges.

Google removes sites themselves, not IP ranges. The only place that this actually comes into play is with RBLs (Realtime Blackhole Lists) and having a Dedicated IP doesn’t get you out of those consequences, either, since your mail comes off the server’s IP regardless of whether you have your own Dedicated IP or not. The way you avoid this is by not hosting with a host that would host the gambling site, porn ring, and Viagra dealer in the first place – or a host that’s vigilant in monitoring and responding to issues on the server.

Another issue to note is that the slower your server, the fewer pages the spiders will be able to index on each visit (they don’t want to crash it). Fewer pages indexed equal fewer pages in the SERP, which decreases your ability to properly theme your site, which in turn will hurt your rankings. I think the connection is pretty clear.

A Dedicated IP will not speed up the server. If it would, we’d bend ourselves over backwards working to get all of you Dedicated Ips so the server would magically hum and purr, rather than, oh, buying really good hardware and tweaking configurations all the time. A Dedicated IP does not equal a Dedicated server, and a Dedicated IP on a shared hosting server is not the web hosting world’s equivalent of a DisneyWorld FastPass.

If you are on shared hosting, you still have to share. Period. If your server is so poor that crawling will slow down or crash the server, that will happen whether you have a Dedicated IP or a shared IP.

If you find out you’re not on a dedicated IP, we recommend calling your hosting company and asking them to switch you over. There may be a small fee, but it’s nominal and is worth the charge.

Any hosting company worth their salt will not provide an IP to you simply because you’ve bought into the myth that somehow Google will find your content far more dazzling simply because you finagled your way into a Dedicated IP. “SEO” is not a justification to ARIN to grant you a Dedicated IP, and considering the run on IP addresses for legitimate purposes and not because they’re seen as fairy dust that will somehow make your site faster, you won’t get one just because you want one.

The best way to get good ranking with Google is the same as it’s always been – create a good site. Have and offer something people want. Create original content that people want to read. Do it the old fashioned way – tweak your meta-tags, submit your URL, and work at it.

The best way to get your PR up is still the hardest – good, old-fashioned work.

I’d like to thank an un-named source at The Google itself that pointed out that Google has, in fact, openly dispelled this myth publicly back in 2002 in a Slashdot Interview with Google Director of Technology Craig Silverstein.

5) Google and IP address.
by Anonymous Coward

Why in this day and age does google continue to penalize sites that are virtual hosted? With ip addresses becoming harder to get/justify every day why does google discount the relevance of links that don’t come from a unique ip address. Please don’t just deny it, I think the Internet community deserves an explanation.

Craig answers:

I can’t just deny it? What are my other choices? [:)] Actually, Google handles virtually hosted domains and their links just the same as domains on unique IP addresses. If your ISP does virtual hosting correctly, you’ll never see a difference between the two cases. We do see a small percentage of ISPs every month that misconfigure their virtual hosting, which might account for this persistent misperception–thanks for giving me the chance to dispel a myth!

Thanks, Mr. Un-Named Source – I love to be able to publish something from 2002 that makes a whole lot of people that have been selling snake oil for 6 years about, especially when I didn’t know that ya’ll had gone on record saying this was BS.

So, there you have it. It’s BS, says the Google.