#WHITELISTed user-agents User-agent: Bingbot disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* User-agent: Googlebot disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* User-agent: Slurp disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* User-agent: Caliperbot disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* User-agent: Mediapartners-Google disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* User-agent: AdsBot-Google-Mobile disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* User-agent: AdsBot-Google disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* User-agent: Googlebot disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* User-agent: AdsBot-Google-Mobile-Apps disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* User-agent: SemrushBot disallow: /app/ad* disallow: /app/pr* disallow: /app/assi* disallow: /app/gu* disallow: /app/ki* disallow: /app/asy* disallow: /app/wi* disallow: /app/po* # block all other undefined bots from crawling User-agent: * disallow: /