# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file Sitemap: https://www.saashub.com/sitemaps/sitemap.xml.gz User-agent: Amazonbot # Amazon's user agent Disallow: /do-not-crawl/ # disallow this directory User-agent: MJ12bot Disallow: /