image_02.jpg
Can You handle their attention ?
Loading...

Can I prevent search engines from crawling scripts such as Javascript and CGI?

When your external scripts like JavaScript and CGI scripts are not protected, they are accessible by Search Engine spiders. This increases Download time as the spiders Crawl your scripts instead of searching for relevant information. The same is true for external CSS style sheets, but because webmasters can use style sheets to “cheat”, the global opinion between SEO experts is that Search engine spiders should have access to CSS style sheets.

 

Solution: Put your scripts in a specially made Directory that is not accessible to spiders. You do this with the Robot exclusion protocol. Open a text writer like notepad and enter the following:


User-Agent: *
Disallow: /scripts/

 

Save these lines as robots.txt and place the file in your root directory. This disallows all (*) spiders (User Agents) access to the /scripts/ directory

However, not all search spiders respect these rules.

0 votes so far for this article. Did this article help you? YES NO
Article Id: 94 - Version: 2 - Created: 29-11-1999 - Last Updated: 29-11-1999 - Hits: 670 
Keywords:
Categories: Do's and Dont's

Comments & Questions

No comments yet!

Try Seo Effect
Free premium seo software plan

Order our Free plan: get acces to all premium features now!

As featured in
Book search engine marketing

Handbook Search engine marketing

RSS feed
  • Subscribe
      textbox
      RSS 0.91
      RSS 1.0
      RSS 2.0
      ATOM 0.3
      OPML
      Subscribe with Bloglines
      Add To Google
      Add To MyYahoo
      Add To My AOL
      Subscribe in NewsGator Online
      Add To Pageflakes
      Subscribe With Pluck RSS Reader
      Subscribe in Rojo
      Add To netvibes
Subscribe