Some times you may want to block Search Engines from indexing your site/blog or personal blog for what ever reason.It is very simple to block search engines from indexing your website by adding a small META code in your web pages or by adding a small text file called robots.txt.
Blocking Search Engines by using method 1 : Method 1 is easy and simple,just you have to add a single META Tag to head of your web pages.
<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
Blocking Search Engines by using method 2 : Create a robots.txt using Notepad. Save it into your root directory of your domain to prevent search bots from accessing any page on your site. Type necessary details into the robots.txt file
To exclude all robots from the entire server
User-agent: * Disallow: /
To allow all robots complete access
User-agent: * Disallow:
To exclude all robots from part of the server
User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /junk/
To exclude a single robot
User-agent: BadBot Disallow: /
To allow a single robot
User-agent: Google
Disallow:
User-agent: *
Disallow: /
Be careful and if you want more details then visit robotstxt.org