The instinct, when you first notice AI crawlers in your logs, is to block all of them. It feels like the safe default. It usually isn't, because "AI crawler" covers at least three different jobs and blocking them has three very different consequences.
Get the distinction right and robots.txt becomes a reasonable tool. Get it wrong and you quietly delete yourself from the places people are starting to search.
Training crawlers collect pages to train models. GPTBot and ClaudeBot are the obvious examples. Blocking these is the closest thing to a pure opt-out: you lose nothing that sends you traffic today, and you keep your writing out of the next training run. If your objection to AI is about your work being used without payment, this is the one you care about.
Search indexers build the index behind AI search answers. OAI-SearchBot and Claude-SearchBot do this. These are much closer to Googlebot in function. Block them and you don't appear in those answers, which means no citation and no click. OpenAI is explicit that sites blocking OAI-SearchBot won't surface in ChatGPT search.
User-triggered fetchers run when an actual person asks. ChatGPT-User, Claude-User and Perplexity-User fetch a page because someone pasted your link or asked a question your page answers. There's no crawl here, no bulk collection, just one request on behalf of one reader. Blocking these blocks people.
Most site owners who say "block the AI bots" mean the first category and accidentally take out all three.
robots.txt matches on the product token, one group per user agent:
# Training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# Google's AI training opt-out (not a crawler; a policy token)
User-agent: Google-Extended
Disallow: /
# Search indexing - think hard before disallowing these
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# User-triggered - a person is waiting
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /Google-Extended is the odd one. It isn't a crawler and never appears in your logs. Googlebot does the fetching, and the token only controls whether what it fetched may be used for Gemini training. So you can't block it by IP, and you shouldn't try.
A reminder that catches people out: the first matching group wins and the groups don't merge. A broad User-agent: * block plus a specific Allow group for one bot does not combine the way you'd hope. Write the specific groups out in full.
Every one of these rules is voluntary. It's a note on the door asking politely. The well-behaved crawlers honour it, which is genuinely most of the named traffic, but nothing in the protocol enforces anything.
That's tolerable for the bots that comply and useless for the ones that don't. Perplexity's crawlers in particular have been reported fetching pages from undisclosed addresses after being disallowed, and once a crawler decides to ignore your file, no amount of editing it will help.
So be clear with yourself about what you're doing. robots.txt expresses a preference. If you need enforcement, that lives at the server or the WAF, keyed on verified IP ranges rather than on a user-agent string anyone can type. We wrote separately about how to verify an AI crawler is genuine, and that verification is the part that has teeth.
There's also an argument for not blocking at all. Being cited in an AI answer is roughly where being on page one of search was fifteen years ago, and the sites that shut the door early may find they've opted out of the channel their competitors are using. That's a business call, not a technical one, but it's worth making deliberately rather than by reflex.
Three things worth confirming after you edit the file.
Fetch yourdomain.com/robots.txt in a browser and read what's actually served. Sites with a CDN or a framework in front of them sometimes serve a cached or generated file that isn't the one in the repo.
Watch your logs for a week afterwards. A crawler that's honouring the file goes quiet. One that doesn't is telling you something the file can't fix.
And confirm the disallowed bots weren't the ones sending you readers. Check your analytics for referrals from the AI products before and after, because the cost of blocking a search indexer shows up as an absence, which is the hardest kind of change to notice.
Does blocking GPTBot remove my content from ChatGPT? No. It asks that your pages aren't used in future training. It doesn't touch anything already in a trained model, and it doesn't stop ChatGPT search from citing you, since that's a different bot. If your goal is not appearing in ChatGPT at all, one robots.txt line won't achieve it.
Should I block CCBot? CCBot is Common Crawl, a non-profit whose archive is used for research as well as by AI companies. Blocking it is a blunt instrument that removes you from a general-purpose public dataset, not only from model training. Worth deciding on its own merits.
Will blocking AI crawlers hurt my Google ranking? Blocking GPTBot or ClaudeBot has no effect on Googlebot, which is a separate crawler following separate rules. Google-Extended is also documented as not affecting search ranking. Just don't let a broad User-agent: * rule catch Googlebot by accident, which is the real risk when editing this file in a hurry.
How do I know a bot is really who it says? You check the connecting IP against the vendor's published ranges, or you run a forward-confirmed reverse DNS lookup with our reverse DNS and DNS lookup tools. The user-agent string on its own is unverifiable.
0 comments
No comments yet. Be the first.