User-agent: BLEXBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Seekport Crawler Disallow: / User-agent: Mediapartners-Google Disallow: User-agent: * Crawl-Delay: 1 Disallow: /job/* Disallow: /holiday/* Disallow: /click Disallow: */page*$ Disallow: *-telesales* Disallow: *3F* Disallow: *search=1* Disallow: *ad=gbase* Disallow: *?a=* Disallow: *&a=* Disallow: *rss=1*