Using the Robots Meta Tag to Reduce Duplicate Search Engine Data
Chris Pearson is a stud when it comes to all things web, not to mention the guy’s relentless pursuit of wordpress goodness keeps me coming back to his blog again and again.
Yesterday morning, a post from him via DIYThemes.com grabbed my attention. It was all about how to use the Robots Meta Tags option in the very popular theme Thesis. While I dont use Thesis, I still found it extreamly interesting and started to investigate its declaration:
One of the biggest problems with WordPress is the fact that it automatically generates different kinds of archive pages that can be indexed by search engines. From date-based archives (daily, monthly, yearly) to tags to categories, these auto-generated pages all contain duplicate content that doesn’t belong in search engines.
Like many times before, he’s exactly right.
Maybe you don’t think its a big deal. Okay, I get that…but think of this:
The job of search engines is to index your site’s pages and determine what your site is about. If you have 100 unique article pages, then ideally, search engines should only have to crawl 100 pages to index your site fully.
However, if you’re using categories, tags, and date-based archives (and depending on how overboard you tend to go with categorization and tagging), then you’re going to have at least one additional page per category, per tag, per month, etc.
Now, instead of having 100 pages to index, you may have 600 pages. Considering you only have 100 pages of unique content, forcing a search engine to index 600 pages to determine what your site is about just doesn’t make any sense.
Think of it this way: Would you rather read a 100-page book or a 600-page book that tells the exact same story? Further, which one do you think you’ll understand better? Which will hold your focus better?
Wow, now that’s heavy.
Functions Hack
So since its implemented with Thesis so beautifully, and since I love to hack my functions.php file…I figured, I’d do just that. I’d put together a super-tiny function that fixes this, albeit small, issue for my specific customization.
1 2 3 4 5 6 7 8 9 10 11 | // META ROBOTS functions.php hack for proper indexing by Search Engines function tq_meta_robots() { // If you would like to disable the ODP or Yahoo! Directory // keep the next two lines. if( is_home() ) echo '<meta name="robots" content="index,follow,archive,noodp,noydir" />'; if( is_archive() || is_tax() ) echo '<meta name="robots" content="index,nofollow,noarchive" />'; } add_action('wp_head','tq_meta_robots'); |
There…that’ll do it.
The first part checks to see if its the home page (since “noodp” and “noydir” are domain specific settings), then sets the proper search engine robot setting to allow indexing, link following, archiving and prevents search engines from pulling extra information regarding your site from the ODP or DMOZ and the Yahoo! Directory.
The second part checks to see if the page is any archive type page (Category, Tag, Author and Date based pages are all types of Archives), or any taxonomy related pages; and sets the information for those types of pages. This sets the search engine robot setting to allow indexing, tells it that links are to NOT be followed, and to not archive the page.
Done and Done.
Previous Post / Next Post Image Thumbnails using Autofocus
So I’ve been uploading photos for a few months now to my photoblog and it’s been interesting. I love the Autofocus theme and all the amazing things it does, but it lacked something…
What’s Missing
My photoblog, that is using the amazing Autofocus theme is simply that…a simple photoblog of some of my photos taken from any and everywhere.
I noticed something wasn’t right and I had one gripe about it…on single pages it lacked a bit of class and touch that I think photoblogs should have. Mainly I was looking for a previous/next thumbnail support when it displayed previous and next navigation links under the content. I thought that would be a perfect place to put these little gems.
By default, Autofocus theme puts the navigation to the previous and next posts underneath your content photo. Makes perfect sense, but I figured it’s a friggin photo blog, why not display the previous/next available photo too? [...]
Embedding RSS Anywhere in WordPress
For some reason or another, I never figured out until today digging through the widgets files in wp-includes directory in your default WordPress installation, that you could easily insert RSS feeds anywhere in your site.
This started with a task I had for MakeUseOf.com to include RSS feeds for the author pages. After digging into it, I found it was quite easy and it uses the Magpierss code almost exactly.
All you really have to do is insert a little code in the template file you want to use (or a page or post using one of many php inclusion plugins for WordPress).
[...]





About TQuizzle
Lover of technology, avid Twitter user, WordPress Wizard and busy husband. With a full-time job, side gigs, being involved with a missional community and starting to build a house, the pressure is on. I say...bring it!
Subscribe via Email