The robots.txt file is very important part of the way your site interacts with other sites, site visitors. and most importantly, search engines. The file dictates what computers can view your site. The reason why it is called the robots.txt file is because it is generally used to block robots from crawling and indexing your site.
Search engine robots crawl the web routinely, indexing every piece of data along the way. If your site is not ready for the search engines then you may want to edit your robots.txt file to block all robots. Some web hosting companies do not include a robots.txt file automatically, so it is up to you to create this basic text file. If you are having trouble creating or using your robots.txt file you should contact you web hosting support staff.