← Check a site

Let AI search bots into your site

Find the robots.txt rules that block AI search bots, and remove them.

Worth

30

points of 100

4 steps, about 10 minutes

0 of 4 done

  1. Open your robots.txt

    It is a small text file at yoursite.com/robots.txt that tells bots which pages they may visit.

    Open your robots.txt

  2. Delete the lines that block AI bots

    Each block is a group: one or more User-agent: lines naming bots, then Disallow: /. Delete the groups that name a search bot or Claude-User.

    If you deleted a group for all bots (User-agent: * with Disallow: /), put this in its place. A bot that no group names is already allowed, so you do not need anything else.

    User-agent: *
Allow: /

  3. Save and publish the file

    Where you do this depends on how your site is built.

    In your Shopify admin, go to Online Store, then click the menu on your theme and choose Edit code. Add a new template, choose robots, make your change and save. Shopify’s guide

    If your site uses Cloudflare, it can add its own rules to the top of your robots.txt. By default these block training bots only, which does not change your score. If the Search or Agent setting in Cloudflare’s AI bot policies is set to Block, it also blocks search bots. Cloudflare’s robots.txt setting

  4. Check it worked

    Run the checker on the same page. Each bot should now say Allowed.

    Bots do not see the change straight away: the robots.txt standard lets them keep an old copy for up to 24 hours. Our check reads robots.txt only, so a firewall that blocks bots without saying so in robots.txt will not show up here.

    Check your site
Why this matters and our sources

Your robots.txt is a small text file at the root of your site, such as example.com/robots.txt. It tells bots which pages they may visit.

We look at it for eight bots. Search bots, which AI tools use to find pages: OAI-SearchBot (ChatGPT), PerplexityBot, Claude-SearchBot, Googlebot (Google’s AI Overviews and Gemini) and Bingbot (Copilot). And bots that fetch a page when a person asks about it: ChatGPT-User, Claude-User and Perplexity-User. The check fails if any search bot is blocked, and needs work if only Claude-User is blocked. ChatGPT-User and Perplexity-User are shown but get no points (see below).

We also show three training bots (GPTBot, ClaudeBot and Google-Extended). Blocking them is your choice and does not change your score.

The AI companies say this in their own documentation. OpenAI says sites that block OAI-SearchBot “will not be shown in ChatGPT search answers”. Perplexity recommends allowing PerplexityBot “to ensure your site appears in search results”. Anthropic says its bots honour robots.txt. So a search bot you block cannot read your page for that tool. That is why this check is labelled Strong.

Two fetch bots may not follow robots.txt at all. OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores robots.txt rules”, because a person started the visit. So blocking them in robots.txt may not stop them, and allowing them proves nothing either. That is why they get no points. Anthropic says Claude-User does follow robots.txt, so it counts.

EvidenceStrongWhat the labels mean

Sources, checked 26 Oct 2026

Every copy-paste fix on this page was put on a test page and passed our own checker before it was published.