- Category
- AI search
- Level
- Intermediate
- Operator
- OpenAI
- Purpose
- Model training
- robots.txt token
- GPTBot
- Check it free
- Robots.txt Checker
What is GPTBot?
GPTBot is the user agent OpenAI uses when it crawls the public web to gather training data for its models. OpenAI introduced it in 2023 and documents that it respects robots.txt.
OpenAI runs other agents for other jobs. OAI-SearchBot finds pages for ChatGPT search results, and ChatGPT-User fetches pages when a user asks ChatGPT to open them.
Should you block GPTBot?
That's a business decision. Allowing it lets future OpenAI models learn about your brand from your own pages. Blocking it keeps your content out of training. Either way, your appearance in ChatGPT search is controlled by OAI-SearchBot, not GPTBot.
How to control it
To block GPTBot from your whole site:
User-agent: GPTBot
Disallow: /To allow it, you don't need a rule, as long as no broader rule blocks it.