A text file containing code that can exclude certain pages or folders from being indexed in the search engines. Can also be used to block access for a particular robot. The robots.txt file must be located in the root directory of the website.
A text file containing code that can exclude certain pages or folders from being indexed in the search engines. Can also be used to block access for a particular robot. The robots.txt file must be located in the root directory of the website.
