XeonBD Official Blog

Robots.txt :: for WordPress

If you are a regular user of our official blog, you might have introduced Mr. Robots.txt before. If you are not, read our archived article The Importance of robots.txt.

After analyzing many well furnished robot for various different sites, I am giving you the best robots.txt for well known wordpress blogging system.

Why I am calling it best! Because it has been made to let the robot find the exact search result of your site, as it never command to search garbage files of your blog which most of the time let the search robot go to wrong ways and consume times as well as sucked up your bandwidth. As a result your site does not get exact search result position.

Here is the robots.txt file you can use it for your wordpress blog.

User-agent: *

Disallow: /cgi-bin

Disallow: /wp-admin

Disallow: /wp-includes

Disallow: /wp-content/plugins

Disallow: /wp-content/cache

Disallow: /wp-content/themes

Disallow: /trackback

Disallow: /*trackback

Disallow: /*trackback*

Disallow: /*/trackback

Disallow: /*?*

Disallow: /*.html/$

Disallow: /page

Disallow: /*feed*

User-agent: Mediapartners-Google

Allow: /

User-agent: Adsbot-Google

Allow: /

User-agent: Googlebot-Image

Allow: /

User-agent: Googlebot-Mobile

Allow: /

#User-agent: ia_archiver-web.archive.org

#Disallow: /

Sitemap: http://www.sitename.com/sitemap.xml

Related posts:

1. The importance of robots.txt
2. Google Site Map Generator is now running on XeonBD cPanel advanced server
3. Why does rip off report rank so high on Google and other search engines?
4. Search Ranking & PageRank