# Basic robots.txt file by Exelwebs.com #Format: # The User-agent allowes or disallows bots and spiders. #The way it is now it will allow all spiders and bot to visit your site. #Disallow is the files and folders you dont want the bots and spiders to follow User-agent: * Disallow: /admin/ Disallow: /shop/index.htm