Some crawlers for search engines obey the Robot Exclusion Standard. We can simply include a "robots.txt" file to a root directory of web servers to tell crawlers what pages to be excluded. The "robots.txt" should look like the following:
For Webmasters and Web Publishers
To protect Web pages from being cached.
Through Meta tags
Another method is to add a NOINDEX tag to web pages those we want to exclude.
For Yahoo and Google, the tag should look like
<META NAME="robots" CONTENT="noindex">
For MSN, the tag should look like
<META NAME="*" CONTENT="noindex" />