Dave Stopher SEO

I have come up against this small probelm lately. I wanted to rewrite one of my sites so that my https pages would not get cached by google.

There is one easy wsay to do this that work for me. What you need to do is first make a second robots file and call it whatever you want for this example I will call it Robots_ssl.txt

With this robots file place the below bit of script.

user-agent: *

Disallow: /

This will prevent any spiders from going on your site.

Now the bit that tell the spiders if they follow this rule or not. This bit of script needs to be placed in the .htacces file on the server. (be warned that this only works on an apache server).

RewriteEngine on
Options +FollowSymlinks
RewriteCond %{SERVER_PORT} ^443$
RewriteRule ^robots.txt$ robots_ssl.txt

This should tell the spiders that if the pages is a https(server 443) then follow the robots file shich is called robots_ssl.txt.

So this therefore will stop any pages from being cached. If you have already got pages cached however you just need to register your https site with google webmaster tools and then you need to remove the site from the listing.