Can I prevent search engines from crawling scripts such as Javascript and CGI?

When your external scripts like JavaScript and CGI scripts are not protected, they are accessible by Search Engine spiders. This increases Download time as the spiders Crawl your scripts instead of searching for relevant information. The same is true for external CSS style sheets, but because webmasters can use style sheets to “cheat”, the global opinion between SEO experts is that Search engine spiders should have access to CSS style sheets.

Solution: Put your scripts in a specially made Directory that is not accessible to spiders. You do this with the Robot exclusion protocol. Open a text writer like notepad and enter the following:

User-Agent: *
Disallow: /scripts/

Save these lines as robots.txt and place the file in your root directory. This disallows all (*) spiders (User Agents) access to the /scripts/ directory

However, not all search spiders respect these rules.