User-agent: * Crawl-delay: 5 Disallow: /api/ Disallow: /html/ Disallow: /scripts/ Disallow: /connect Disallow: /forgot Disallow: /login Disallow: /profile_loader Disallow: /redirect Disallow: /remote_file Disallow: /review Disallow: /screenshot Disallow: /signup Disallow: /site_review Disallow: /*?action=* Disallow: /*?*&action=* Disallow: /*?request=* Disallow: /*?*&request=* Disallow: /*?sort=* Disallow: /*?*&sort=* Disallow: /*?sorting=* Disallow: /*?*&sorting=* # Disallow bad bots User-agent: 008 User-agent: AhrefsBot User-agent: Amazonbot User-agent: archive.org_bot User-agent: Baiduspider User-agent: Barkrowler User-agent: BLEXBot User-agent: BecomeBot User-agent: CCBot User-agent: dotbot User-agent: EtaoSpider User-agent: Exabot User-agent: MJ12bot User-agent: NPBot User-agent: OmniExplorer_Bot User-agent: Rogerbot User-agent: SearchmetricsBot User-agent: SeekportBot User-agent: SemrushBot User-agent: SemrushBot-BA User-agent: SemrushBot-BM User-agent: SemrushBot-CT User-agent: SemrushBot-SA User-agent: SemrushBot-SEOAB User-agent: SemrushBot-SI User-agent: SemrushBot-SWA User-agent: SentiBot User-agent: SEOkicks User-agent: SEOkicks-Robot User-agent: SeznamBot User-agent: ShopWiki User-agent: sistrix User-agent: SISTRIX Crawler User-agent: SISTRIX User-agent: SMTBot User-agent: sogouspider User-agent: Sogou web spider User-agent: Stanford User-agent: Stanford Comp Sci User-agent: Synthesio User-agent: UnisterBot User-agent: YandexBot Disallow: /