Free tool

Robots.txt checker for Google and AI crawlers

See exactly what your robots.txt lets Google, Bing and AI crawlers fetch, which rule decides it, and what's wrong with the file. Edit the rules below and test them before you change your live file.

We test the page you entered, or this path on the same site.

What it checks

Test what your robots.txt lets 22 crawlers fetch, including Googlebot and the bots behind ChatGPT, Claude and Perplexity.

  • Allowed or blocked, with the deciding rule, for 22 crawlers: search engines, AI answer crawlers, AI training crawlers and SEO tools
  • Matching the way Google and RFC 9309 describe: the most specific group, then the longest rule, with wildcards and $
  • Problems in the file: unsupported lines such as noindex, rules before any user-agent, full URLs in paths, size over Google's 500 KiB limit, and server errors
  • Your sitemaps, and an editor to test changes against any path before you publish them

Frequently asked questions

Does blocking a crawler in robots.txt remove my pages from search?

Not by itself. robots.txt stops crawling, not indexing: a blocked page can still appear in Google if other sites link to it. To keep a page out of results, let it be crawled and add a noindex tag.

Should I block AI crawlers?

It depends on what you want. Blocking answer crawlers such as OAI-SearchBot or PerplexityBot keeps you out of those assistants' answers. Blocking training crawlers such as GPTBot or Google-Extended keeps your content out of future models but doesn't stop you being cited. Most brands that want to be recommended allow the answer crawlers.

Why does my robots.txt allow a crawler that still can't reach my site?

robots.txt is only one gate. Firewalls, CDN bot protection and some hosts block crawlers at the server. Our AI Visibility Check requests your page as each crawler to catch that.

Related free tools

All free tools

Want us to fix what the tools find?

Book a callFree AI visibility check
Chat on WhatsApp