Should I Block or Allow AI Crawlers Like GPTBot and Google-Extended?
You received an email from someone with a blocking AI-bots article. Now it’s time to make a decision you weren’t given the price of. The question is easy to type: “should I block AI crawlers?” It is hard to answer. There is no one right answer, and this guide will not pretend there is. It tells you what each crawler does and what it removes you from, and provides you with a 5-question decision tree that you can follow in a few minutes.
This guide explains:
- The three kinds of AI crawlers and why the difference matters
- Why blocking Google-Extended does not remove you from AI Overviews
- The honest case for blocking
- The honest case for allowing
- A decision tree you can walk in five minutes
Should I block or allow AI crawlers?
It’s up to your content as it determines whether it’s the product or the advert for the product, because blocking protects what you sell, and it also takes you away from the answers you would have generated.
This guide does not pick a side. For different businesses, both choices are fair, and outside your business, advice would be a guess. Most articles about “how to block AI bots” don’t mention the cost, and most articles about “how to allow” don’t mention the risk.
You can use the decision tree below to help you make your decision.
What are these crawlers actually doing?
Three different things: collecting content to train a model, fetching pages to answer a live question with a citation, and loading one page because a user asked for it in a chat.
Content is gathered by the training crawlers and can be used as training data for a model. Retrieval crawlers retrieve pages to answer a real-time query and identify the source of the page. Fetchers load one page when a person requests it in a chat.
As noted in the OpenAI docs, there are separate crawlers for each of these jobs, and the user agent names and purposes in the table below are directly from OpenAI’s GPTBot documentation, not from a blog, so you can check each claim at the source and see for yourself when it last changed.
The other rows are based on the public description of each operator. If you wish to obtain a full “AI crawler list”, you should be aware that there is no such thing and that names are subject to change.
| Crawler | Operated by | What it is documented to feed | What blocking it removes you from |
|---|---|---|---|
| GPTBot | OpenAI | Model training and improvement | Future training use. It does not by itself remove you from live answers |
| OAI-SearchBot | OpenAI | Search and retrieval for answers | Being found and cited in live ChatGPT search answers |
| ChatGPT-User | OpenAI | Loads a page because a user asked for it in a chat | That user’s own request for your page. OpenAI notes robots.txt rules may not apply to these visits |
| Google-Extended | Use of content for Gemini and related generative products | Gemini training and grounding. Not Google Search, and not AI Overviews | |
| Googlebot | Google Search index, which AI Overviews are built from | Google Search entirely. Almost nobody should do this | |
| PerplexityBot | Perplexity | Retrieval and citation in Perplexity answers | Citations in Perplexity |
| ClaudeBot | Anthropic | Crawling for Anthropic’s models and products | Anthropic product exposure |
| Applebot-Extended | Apple | Generative model use, separate from Apple’s search crawler | Use of your content in Apple’s generative features, not Apple’s search results |
| CCBot | Common Crawl | A public dataset many parties train on | A dataset you cannot audit, used by parties you will never know |
The 9 rows are not as important as the three types. A business that blocks retrieval crawlers and user fetchers when it meant to block training has removed itself from live answers by mistake.
The risk is obvious to an online store, as a training crawler can read all product descriptions, prices and images. Weighing that against the product visibility that an assistant can provide, our work on product data exposed at scale. A catalogue is then protected where it should be and is found where it should be.
Why Google-Extended is not the AI Overviews switch
Google-Extended is for the use of content in Gemini and other generative products, and AI Overviews are derived from content in the Google Search index that Googlebot collects, so that blocking Google-Extended does not exclude you from AI Overviews.
In Google’s own words, robots.txt rules for Googlebot control how a site is crawled for Search, while Google-Extended is for limiting AI training and grounding in some of Google’s other systems. Google’s documentation on AI features and your website states this directly. That is why there is a “google-extended robots txt” rule that changes the way Gemini uses it, but not Search.
The conclusion is clear: If you want to completely avoid AI Overviews, you’ll need to discontinue Google crawling your site for Search, which is something few businesses would want. Google provides a preview of nosnippet, data-nosnippet, max-snippet and noindex controls for owners looking to show less. These also affect the interpretation of normal results.
The case for blocking
When the written work is the asset, when an answer that summarizes your page is a substitute for a visit, or when you don’t want your work used without your credit or payment, then blocking is the fair choice.
Publishers, Course sellers, Template sellers and Companies that research and sell research have the best case. A summary that satisfies the reader means no visit, no subscription and no attribution, and a training run takes the work with none of the three. It is a fair choice for them to make to opt out.
The protection of the content should be commensurate with its effort, and the first step is determining what the asset is. Our approach to work that took real effort to produce starts by separating those pieces from the ones that only advertise. The strongest controls are only necessary for the first group.
The same logic applies to professional services. A company that produces the research reports and detailed guides can justly conclude that these should not be used as a model. Our work for firms whose written work is the asset treats that choice as a business decision to be written down, not a default, so the reason can be revisited when the tools change.
The case for allowing
The fair choice is to allow when being found is more important than control, which is the case for most local and service businesses, as an assistant who can’t read your pages will suggest a competitor who can. If you allow it, let an assistant read, fetch and cite you, which is the initial condition for being named at all. Our view of where crawler policy fits the strategy is that access comes before every other search effort. So the decision belongs inside your organic plan, not beside it.
The visibility is real, and so are the limits. The SEO ranking signals “allow AI crawlers SEO” imply this, but Google will rank content based on the people-first approach outlined in Google’s guidance on creating helpful content. The guidance is not about access to the crawler, but about the quality of the content, so that access to the crawler is a prerequisite for being cited and not a ranking technique. Be honest about the return: the clicks an answer sends back are often few, and a named mention has a value that is hard to price. Our guide to what access makes possible explains what an assistant checks before it names a business. None of that matters if it cannot read your pages.
The decision tree
Walk through five questions in order, stop at the first that describes your business, and write down the reason, because the worst version of this decision is the one made by default and forgotten.
The table below is the tree. Stop at the first row that describes you.
| Question | If yes | If no |
|---|---|---|
| 1. Is your content the product, sold or licensed as the thing people pay for? | Block training crawlers. Think about allowing retrieval crawlers so buyers can still find you | Go to 2 |
| 2. Do you need customers to find you through an assistant? | Allow retrieval crawlers. Decide on training separately | Go to 3 |
| 3. Is your material private, regulated or client-specific? | It should not be publicly crawlable by anyone. Fix access, not robots.txt | Go to 4 |
| 4. Does your traffic depend on informational content that an answer could replace? | Allowing costs you something real. Weigh it against being absent from the answer | Go to 5 |
| 5. Are you a local service business selling in one area? | Allow. There is little to protect and a named mention is worth having | Make the call on purpose and write down the reason |
Once you have a row, the work is setup, not opinion, and the main risk is a typo that blocks the wrong crawler. Our technical team handles crawler access configured deliberately. That includes reading what your file says today, writing the rules and testing that the pages you want crawled still are.
How do I actually do it?
You set access per user agent in robots.txt at the root of the domain, with one group per crawler, and the operator, not any article, publishes the right name for each.
People search “block GPTBot robots txt” and expect one line to settle it. The example below is for illustration only. You must check every name against OpenAI’s GPTBot documentation and each other operator’s own page before you use it, because names change without notice and a mistyped one fails silently without any warning.
# Illustrative only. Verify every token before use.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
The file lives at the root and applies per host and per protocol. Staging and production often differ, so check the live file. The most common failure is a blanket disallow directive written for one crawler that also catches Googlebot, which removes the site from Google Search.
On WordPress, the file is often made on the fly or edited through an SEO plugin, so changing the wrong copy has no effect. Our work on editing robots and headers on WordPress confirms which copy the live site serves and tests the result before and after the change, so you know the rule is doing what you meant.
What robots.txt does not do
The file is a request to crawlers that choose to follow it, it has no effect on content already collected, and it does nothing about a person pasting your page into a chat.
Four limits apply. Robots.txt is a request, not an enforcement tool, so a crawler that ignores it is unaffected. Blocking today does not remove content already used.
A block does not stop a person pasting your page into a chat. The list of crawlers also changes faster than any published article can track.
Robots.txt is not a security control, and content that must not be read should not be public at all. This article is not legal advice. A different file often gets mixed up with it, and our guide to the file people confuse this with explains why that separate proposed convention does not grant or deny crawler access the way robots.txt does, so do not treat one as a substitute for the other.
How would I know what the decision cost me?
Record a baseline of assistant referrals and manual prompt results before you change anything, then recheck on the same schedule, because a change with no baseline cannot be judged.
Without that baseline, you cannot tell a cost from a coincidence, and a change that looks harmless on day one can show up as missing referrals three months later. Our guide to watching what the decision costs explains how to find visits from assistants in your analytics reports and how to run the manual prompt check on a schedule.
Deciding whether to block AI crawlers, in order
The order is: find out what your file says, split the crawler types, walk the tree, check every name, set a baseline and put a review date in your diary.
1. Find out what your file says now.
2. Separate training, retrieval and user fetches.
3. Walk the five questions.
4. Check every name against the operator’s own page.
5. Set a baseline before you deploy.
6. Put a review date in your diary.
Whether you ask “should I block AI crawlers” or not, the one firm point is that the decision should be on purpose, because most sites have never looked at the file. Want to see how this works for your site? If you would like a developer to read your file with you, Pink Socials will show you what it says today and which pages each rule affects, then help you decide which crawlers to allow and how to record the reason.
Leave a Reply