SEO Tip of the day
Create a Robots.txt File
Most search engines obide by the Robots.txt protocol which is designed to tell search engines where they are allowed to visit. All search engines that obide by the robots.txt protocol look for the robots.txt file first.
This is in the following location on the root of your domain - www.domain.com/robots.txt - for example, view the robots.txt file on this site - www.theseofiles.co.uk/robots.txt. You do not even need to put anything in the file.
Simply uploading a blank text file named robots.txt is sufficient. This prevents search engines from generating a 404 error and tells them that they are ok to go and spider your site.
If you have some sections, eg an admin directory, that you don't want to be indexed you can tell search engines where they can and can't go with the robots.txt file in the following manner;
User-agent: *
Allow: /searchhistory/
Disallow: /news?output=xhtml&
Allow: /news?output=xhtml
Disallow: /search
This is a short excert from www.google.co.uk/robots.txt
The SEO Files Recomends
Upload a blank robots.txt file so that search engines know they are OK to view your site and minimise 404 errors in the process!