Open your server logs today and you'll find GPTBot, ClaudeBot and PerplexityBot in there alongside the search engines. Some of those hits are genuine. Some are scrapers wearing a costume, because a user-agent string is just text a client sends, and anyone can send anything.
That matters the moment you decide to treat AI crawlers differently from everyone else. Whitelist "GPTBot" by name and you've built a door that opens for anybody who knows the password. Here's how to check properly.
A user agent is a header. It's typed, not issued. There's no signature on it, no certificate, nothing that ties the string to the company it names. I can send a request claiming to be GPTBot in about two seconds with curl, and your logs will record it exactly as if OpenAI had sent it.
So any rule keyed only on that string is decorative. If you're blocking by user agent you're blocking the polite bots, which were going to respect your robots.txt anyway. If you're allowing by user agent, you've made impersonation worth the effort.
What you actually need is the network layer. Where did the connection come from? That's much harder to fake, because the answer has to be routable back to the sender.
The major AI vendors publish the address ranges their crawlers come from, as machine-readable JSON you can fetch and diff on a schedule.
openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json and openai.com/adsbot.json.claude.com/crawling/bots.json, a list of CIDR prefixes with a creationTime on it. The individual bots (ClaudeBot, Claude-User, Claude-SearchBot) share the infrastructure and are told apart by user agent.perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.gstatic.com/crawling/ipranges/, including common-crawlers.json and user-triggered-fetchers.json.The check is simple: does the connecting IP fall inside one of the published prefixes for the bot it claims to be? If it doesn't, the user agent is lying.
Two things to watch. These lists change, so fetch them on a cron rather than pasting a snapshot into your firewall config and forgetting about it. And they're per-vendor, so a range check only answers the question for bots whose vendor publishes ranges at all.
The older technique, and the one Google has recommended for Googlebot for years, doesn't need a list at all. It's two lookups, and it works because it requires control of something an impersonator doesn't have.
Step one. Take the IP that hit you and do a reverse DNS lookup on it. You're asking what hostname that address claims. For Googlebot you should get something shaped like crawl-66-249-66-1.googlebot.com. Other Google patterns include geo-crawl-*.geo.googlebot.com and rate-limited-proxy-*.google.com. You can run this with our reverse DNS lookup.
Step two, and this is the one people skip. Take that hostname and resolve it forward, back to an IP address. Then check it matches the address you started with. A DNS lookup will do it.
Why both directions? Because reverse DNS is controlled by whoever owns the IP block, not by Google. Someone can point their own address's PTR record at crawl-something.googlebot.com and the first lookup will happily return it. What they can't do is make googlebot.com resolve that name back to their address, because that zone isn't theirs. Forcing the round trip to close is what makes the check meaningful.
Confirm that the suffix is right too. crawl-1-2-3-4.googlebot.com.attacker.example contains the string "googlebot.com" and is not Google. Match on the domain ending, not on a substring.
Use the IP ranges where the vendor publishes them. It's one lookup against a local set, it's fast enough to run inline on every request, and it needs no DNS round trip.
Use forward-confirmed reverse DNS when there's no published list, when you want a second signal, or when you're writing something generic that has to handle crawlers you haven't heard of yet. It costs two DNS queries, so cache the verdict per IP rather than resolving on every hit.
For most sites, honestly, the useful version of this is not blocking at all. It's knowing. Run the check over last month's logs, count how much of your "AI crawler traffic" fails verification, and you'll know whether you have a real bandwidth problem or a reporting artifact.
Does a failed check mean someone is attacking me? Usually not. A lot of failures are research scrapers, SEO tools and hobby projects that copy a well-known user agent because it gets fewer doors slammed. It's dishonest but rarely hostile. Treat it as unverified traffic and apply your normal rate limits rather than assuming an attack.
Can I just block every IP that fails? You can, but be careful with user-triggered fetchers. When someone pastes your URL into a chatbot and it fetches the page, that request may come from a different range than the crawler, and it represents a real person trying to read your site. Blocking it blocks them. Check the vendor's separate list for user-initiated traffic before you write a blanket rule.
How often do the published ranges change? Often enough that a hardcoded copy will rot. Anthropic's feed carries a creationTime so you can tell how fresh yours is. Fetch on a schedule, and log when the set changes so a surprise reshuffle doesn't quietly lock out a legitimate crawler.
Is any of this in the HTTP request itself? No, and that's the whole problem. Nothing in the headers is verifiable. Every reliable signal here comes from the network address and DNS, which is why the check lives outside the request. If you want to see exactly what your own browser sends, our user-agent viewer shows it.
0 comments
No comments yet. Be the first.