How to use Robot tags?

Meta Robots Tags are HTML code like this:

<Meta name="ROBOTS" content="Index, FOLLOW">

This type of {ShowGlossar:301} specifically directs robots to index the content of that web page.

If you have this Robot tag <Meta name="ROBOTS" content="NOINDEX, Nofollow"> somewhere in your HTML code then it is strongly recommended that it be removed immediately.

You may use this type of Tags on pages that you do not want search engines to index or spider.

Even better is to not use any robot tag at allÂ…better use one robots.txt text file in the root of your webpage. A search robot that respects this robot rules will always look for this robots.txt file in the root. This allows you to disallow or allow the robots to index whole sections at ones.

Solution: Make a Directory that is not accessible to the spider. You do this with the Robot exclusion protocol. Open a text writer like notepad: put in the following:

User-Agent: *

Disallow: /scripts/

Save this lines as robots.txt and place the file in the root directory. This disallows all (*) spiders (User Agents) to get into the /scripts/ directory