Robot For WordPress To Prevent Duplicate Content

Blog that powered by WordPress always create a lot of duplicate content to be indexed in search engine. Sometimes you can even found your /feed/ page searched in Google search. There are also other duplicate page like feed, trackbacks, comments, categories, and etc. As we know, Google hates duplicate content very much, you’re in deep shit if you dare to do so.

What Is Robots. txt

Robots. txt is created to control Search Engine Bots like Google Bots, Yahoo Bots, MSN Bots and etc. Robots. txt and Search Engine Bots are like Master and Servant, Master can tell Servant what can do and what shouldn’t do. Robots. txt can ask Search Engines Bots what it can index and what it can’t index. In order to control Search Engine Bots, we have to create a txt file and type in some simple command. You’re telling search engine bots that what it’s allowed to index and what it’s disallowed to index.

User-agent: * Disallow: / Allow: /

How To Create Robots. txt

1) Open a txt file.
2) Type in the command as been given above.
3) Save as “robots. txt”.
4) Copy it into your wordpress root folder.
5) That’s It.
6) Sample.

Sitemap: http://www.myblog2day.com/sitemap.xml

User-agent: *
Disallow: /cgi-bin/
Disallow: /css/
Disallow: /feed/
Disallow: /trackback/
Disallow: /k/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /comment-page-1
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments

# Google Image
User-agent: Googlebot-Image
Allow: /*

# BEGIN XML-SITEMAP-PLUGIN
Sitemap: http://www.myblog2day.com/sitemap.xml.gz
# END XML-SITEMAP-PLUGIN

You can simply copy and paste it into your robots. txt, you can add in the file that you don’t want search engine bots to index. Now you’ve created a robots. txt to control search engine bots to index your article, so that it can reduces duplicate content within your blog. By managing your blog well, your articles’ ranking will be obviously maintaining or even increasing to higher rank.