In short
- Three kinds of visit
- Collecting for training, maintaining an index, and fetching live during an answer. Only the last one corresponds to a conversation happening now.
- Verifiable
- Some crawlers publish IP lists or can be verified through DNS. The rest only state a name, and anyone can type that.
- What your log does say
- Which pages get fetched and which never do. That is a list of work, and it is yours alone.
Who comes past, and whose crawler is it?
Below are the crawlers that feed AI answers, per operator, with the question that actually matters attached: can you verify it really was one? For about a third the answer is no, and that is not a detail — a name anyone can type is no proof of a visit.
Note the split at one operator: the crawler maintaining the index is not the one fetching live during a conversation. If you only see the first in your log, you are being collected but not quoted.
| Operator | Crawlers | Verifiable? |
|---|---|---|
| Google-Extended · Googlebot · GoogleOther | yes — reverse DNS on the published domain | |
| Microsoft | Bingbot | yes — reverse DNS on the published domain |
| Apple | Applebot-Extended · Applebot | yes — reverse DNS on the published domain |
| OpenAI | GPTBot · OAI-SearchBot · ChatGPT-User | yes — against the published IP list |
| Anthropic | ClaudeBot · Claude-User · Claude-SearchBot · anthropic-ai | yes — against the published IP list |
| Perplexity | PerplexityBot · Perplexity-User | yes — against the published IP list |
| Amazon | Amazonbot | no — only the name the visitor states, so it can be faked |
| Meta | Meta-ExternalAgent | no — only the name the visitor states, so it can be faked |
| ByteDance | Bytespider | no — only the name the visitor states, so it can be faked |
| Cohere | cohere-ai | no — only the name the visitor states, so it can be faked |
| DuckDuckGo | DuckAssistBot | no — only the name the visitor states, so it can be faked |
| You.com | YouBot | no — only the name the visitor states, so it can be faked |
| Diffbot | Diffbot | no — only the name the visitor states, so it can be faked |
| Timpi | Timpibot | no — only the name the visitor states, so it can be faked |
How do I read my own log without a developer?
Most hosting packages let you download the raw access logs, and then this is a matter of searching for the names above. If your site sits behind a service that filters traffic, the overview is often already there under a heading like bots or verified crawlers.
What you are looking for is not the count but the spread: which pages get fetched. A crawler visiting only your homepage and your contact page has never seen the rest of your site — and that explains an answer in which you exist but nothing specific is said about you.
Why do I see so little AI traffic in my statistics?
Because being named often produces no click. The answer contains your name, the asker has what they needed, and nothing gets clicked. That is not a failure of measurement but the nature of the channel: the gain is that your name was said, not that a visitor arrived.
What you do see in your statistics is the traffic that clicks through after such an answer anyway, arriving with a recognisable origin. That number is almost always smaller than the number of times you were named — and it is precisely that gap that makes measuring both worthwhile.
Should I keep some of these crawlers out?
That is a choice, not a technical question, and it splits in two. Excluding collection for training while still wanting to appear in answers is possible at some operators and not at others — for some it is the same crawler.
The practical advice: exclude nothing until you know what is coming in. Whoever blocks first and measures after is measuring their own setting.
Frequently asked questions
My host does not give me logs. Now what?
Then approach it from the other end: ask the same thing of several assistants and see whether they quote content that only exists on a deep page. If they only know your homepage, nothing further was ever looked at. That is less precise than a log and it costs you nothing.
Does lots of crawler traffic mean I am doing well?
No. A visit means you are read, not that you are chosen. It is a necessary condition and a poor measure: a site fetched in full every day and appearing in no answer at all exists — and that case usually points at what is findable about you outside your site.
Read on
Free, no account and no card.