# Google and Bing must have their own entries or they get the crawl-delay also User-agent: * Disallow: /searchpagination Disallow: /rvdetails/ # Limit crawl to once every thirty seconds Crawl-delay: 30 User-agent: GoogleBot Disallow: /searchpagination Disallow: /rvdetails/ User-agent: Bingbot Disallow: /searchpagination Disallow: /rvdetails/ User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: /