Free tool by 313 Automations

AI Crawler robots.txt Checker

Paste your robots.txt to see which AI crawlers can read your site: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more. Find out whether you are blocking AI search by accident, then generate a policy that keeps you visible in AI answers.

1. Paste your robots.txt

Open yourdomain.com/robots.txt in a new tab, copy everything and paste it here. Nothing is uploaded or stored: the check runs in your browser.

Page path to test Use / for the whole site, or a path such as /blog/

2. Crawler by crawler

Show

Rules are matched the way major crawlers read robots.txt (RFC 9309): the most specific user-agent group wins, otherwise the * group applies; within a group the longest matching rule wins and Allow wins a tie. * and $ wildcards are supported.

3. Build a recommended AI crawler policy

Pick a stance. The snippet updates as you choose. Add it to your robots.txt as its own block, separated from other groups by a blank line.

Get cited, opt out of training
Allow AI search crawlers and user fetchers, block crawlers that collect data for model training. The most common choice for businesses that want AI visibility. Allow everything
Maximum visibility. Your public pages may also be used to train AI models. Block AI, keep classic search
Block AI training and AI search crawlers. Google and Bing search stay open. You will likely disappear from AI answers that cite sources.

Blocking Google-Extended or Applebot-Extended does not affect Google or Apple search rankings. User-triggered fetchers may ignore robots.txt because a person asked for the page.

Want help with AI search visibility?

313 Automations runs GEO audits that check robots.txt, schema, content and whether ChatGPT, Perplexity and Google AI Overviews mention your business.

Get a free GEO check

Crawler list: AI Crawler User Agents dataset (sources linked per row, checked 20 Oct 2026).