Cloaking, flash, shockwave, javascripts and get crawledby the spiders

Get crawled - Other indexing pitfalls

Flash and Shockwave

Search engines do not read Flash and Shockwave files. Content and links in any of these formats will not be accessible by search engines, although Google can now follow links in Flash.
Search engines basically only read regular text on a web page. They don't read text and follow links within special formats such as Java Applets or any other formats that require the user to have a certain program installed.
Read more about Flash, Macromedia and Java Applets in the Links and Navigation section.

Image maps or navigation buttons

Not all search engines read image maps. Most search engines read navigation buttons. If you use image maps you should add the same links as you have in the image maps as regular text-based links.
Read more about image maps and navigation buttons in the Links and Navigation section.

JavaScript and DHTML navigation

Search engines do not read JavaScript. You should not put important content or navigation inside JavaScript unless you also have the same links and content duplicated as regular text and text-based links.

Cookies

Spiders do not read cookies. Therefore, if your web site uses cookies, make sure that all the web pages you want indexed are accessible without accepting the cookie. Par example, if you force a user to set a cookie before loading a page, the search spiders cannot read your pages; they cannot set cookies. So make the cookie set optional.
A cookie is a small text file that a web server can save in a user’s browser for future retrieval when that same user returns. It can be used to store login information or other preferences to make it easier or quicker for returning visitors to use a particular web site.
Cookies are very safe to use in the sense that they cannot be read or shared by different users or across different web sites. Information, like passwords in cookies, can also be 128 bits encrypted, and very secure. If a cookie is set on your browser, the web site that set it is the only web site that can read it. In addition, no other user of the web site has access to the information in your cookie. Cookies can't be transferred or shared.

Tip: test your site with cookies off
You can turn off cookies in your own browser to test if your web site can be accessed without them. Refer to your manual or help files of the browser you are using for instructions on how to turn off cookies. Usually these instructions will appear under "advanced options". For Microsoft Internet Explorer go to tools> Internet options > privacy > block all cookies.

IP delivery, agent delivery, cloaking, and personalization

It has become more and more common for web sites to implement some degree of personalization, regional targeting, or other form of individualization in the way web pages are served to individual users.
In its most simple form it can be a program on the Server that checks what browser people are using, and serves up a specially tailored version for that browser. The same kind of program can also be used to check if people have specific plug-ins installed so that they get a version of the web site they can read.
A more advanced use would be to check what country the user comes from to serve a local version of the page. Some portals and Search engines, as well as many other cross-national web sites do this. Legitimate reasons for using this technique include business strategies, marketing, and legal issues with products that are only allowed in certain countries or localities. However, the
same techniques can also be used to serve different content to Search Engine spiders and normal human visitors. These techniques are usually referred to as "Cloaking". In most cases, search engines do not like the use of cloaking. Your pages can be banned by using cloaking! Thus, do not use cloaking unless you have a legitimate reason to do so.