Many bloggers do not use robots.txt file at all. Even if they use robots.txt, there are only few lines of code to restrict search engines bot. As many of you already know robots.txt file analysis is one of the essential factor for search engine optimization. Google also included this tool inside Google Webmaster tools to encourage webmasters to use this feature. Now, i will explain how you can use this robots.txt and decrease your search engine supplementary results and increaser rankings (both for bloggers and normal websites).

First of all, what is robots.txt file? And what does it do for a website or blog anyway?

robots.txt is used for showing search engines bot (like Google, Yahoo, MSN search bot) what to crawl or index from your blog or websites and what not to.

So, if you use robots.txt you can decrease the amount of duplicate content that are picking up by the search bots and placing them to supplementary index. Supplementary index or duplicate content means those pages or content that search engines are finding twice when they are visiting your blog. If you notice carefully there are actually many pages that are duplicating your blog content. Specially your rss feed, trackback URL, categories, blogs pages (pagination) etc.

After search traffic has been dropped last week, i made a post about supplementary results and how it is fictional. Well, just to be on the safe side i added some commands to my robots.txt file and surprisingly within few days my supplementary results decreased. The traffic has not raised that much yet but it’s increasing bit by bit. So, i would recommend all fellow bloggers to try this method and see if it works for them too.

After you configure you robots.txt file, make sure you check if you blocked the right directory or not. Now, you do not have to wait for bots to visit your page so that you can see which pages were blocked and which was crawled. You can do all these analysis in a second using Google Webmaster tools. Here is a screenshot on how to test your robots.txt file commands using Google webmaster tool.

• Login to Google Webmaster Tool
• Click on your blog or sitemap
• Click on robots.txt analysis from the left menu bar and scroll down to see this new feature

Although supplementary results are becoming less important to search specialist like Matt Cutt (many of them believe soon Google will be supplementary results query free), it’s always good to stay on the safe side as i mentioned earlier.