Skip to content

ResourcesAEO Handbook

AI crawler tracking: what it shows, and whether AEO platforms include it

A crawler log shows which AI bots read which of your pages. It never shows whether an answer cited you, and vendors put it on very different plans.

Jack F, Co-founder

MeasurementAI crawlerscrawler trackingGPTBotLLM analyticsserver logsAEO

Part of the Answyn AEO Handbook.

Now readingWhat it shows

AI crawler tracking records which AI bots fetched which of your pages, and whether each fetch was a training crawl, a search crawl or a live fetch for a user's question. It cannot show whether an answer cited you. Where AEO platforms offer it, it usually comes in the plan rather than as a paid add-on, but which plan varies: on 4 November 2026, Peec AI had it on every plan and Otterly.AI from Standard upwards.

Crawler tracking answers an access question: did the AI companies' bots reach your pages, and which ones? Whether their answers then cite or name you is a visibility question, and it needs different data.

We track the buying question behind this page in Answyn: whether AI visibility tools include crawler tracking as standard or sell it as an add-on. In 63 answers collected in the UK between 6 October and 4 November 2026 (ChatGPT, Google AI Overviews and Claude), none named Answyn, and Google's AI Overviews named Semrush in 22 of 28 answers. The table further down sets out what each vendor's own pages say.

What does AI crawler tracking show?

AI crawler tracking is a record of which AI bots requested which pages on your site, taken from the requests your server or CDN receives. Each request names the bot in its user agent, so the record shows who fetched which page, when, and what response the bot got.

The bots do three different jobs, and the job decides what a fetch means. A training crawl collects pages that may be used to train future models. A search crawl builds the index an assistant searches when it answers. A live fetch happens while someone waits: the assistant visits the page because that person's question led it there.

From each company's own crawler documentation, read 4 November 2026
CompanyTrainingSearch indexLive fetches at a user's request
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
PerplexityNone documentedPerplexityBotPerplexity-User
GoogleGoogle-Extended, a robots.txt token with no user agent of its ownGooglebot, for Search, AI Overviews and AI Mode includedGoogle-Agent and other user-triggered fetchers

OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models, and that disallowing it "indicates a site's content should not be used in training generative AI foundation models". OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features", and sites that opt out of it "will not be shown in ChatGPT search answers", though they can still appear as navigational links. ChatGPT-User visits a page when someone asks ChatGPT or a custom GPT a question; it "is not used for crawling the web in an automatic fashion", and because a user starts the action, "robots.txt rules may not apply". A fourth agent, OAI-AdsBot, checks the safety of pages submitted as ads on ChatGPT (OpenAI's overview of its crawlers, read 4 November 2026).

Anthropic draws the same lines. ClaudeBot collects web content that could contribute to training its models. Claude-SearchBot "navigates the web to improve search result quality for users", and Claude-User may visit websites when someone asks Claude a question; blocking it "prevents our system from retrieving your content in response to a user query" (Anthropic's article on its crawlers, dated 7 April 2026).

Perplexity documents two bots. PerplexityBot "is designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models". Perplexity-User "supports user actions within Perplexity", and since a user asked for the page, it "generally ignores robots.txt rules" (Perplexity's crawler documentation, read 4 November 2026).

Google documents no separate crawler for AI Overviews or AI Mode. Rules for Googlebot "affect Google Search (including Discover and all Google Search features)". Google-Extended controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI; it "doesn't have a separate HTTP request user agent string", and it "does not impact a site's inclusion in Google Search". Google also lists user-triggered fetchers, such as Google-Agent, which agents on its infrastructure use "to navigate the web and perform actions upon user request" (Google's list of common crawlers and its list of user-triggered fetchers, read 4 November 2026).

One thing only a log can show is a fetch your robots.txt did not stop. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Google says the same of its user-triggered fetchers. Anthropic says its bots honour robots.txt.

What can crawler tracking not tell you?

It cannot tell you whether an answer cited or named you, because a crawl is not a citation. A log line records details such as the path, the time, the status code and the user agent; nothing in it says which question led to the fetch, what the answer said, or whether it linked to you. Being read is not the same as being recommended, either. In Answyn's own tracking of its category (25 questions, 1,575 answers, UK, 6 October to 4 November 2026), ChatGPT retrieved a page of ours in 49 runs across five questions and still left Answyn out of the answer.

That gap is why the Pages screen in Answyn has a "Crawled, never cited" filter: pages an assistant fetched and still did not cite in the selected window. The models can reach those pages and have read them, and the pages are not persuading them.

It cannot isolate Google's AI features either. With no separate crawler documented for AI Overviews or AI Mode, a log cannot tell a Googlebot fetch for an overview from one for the ordinary results, and Google-Extended never appears because it has no user agent of its own. For Google, Search Console's Generative AI performance report is the better source: it counts how often links to each of your pages were shown in AI Overviews and AI Mode.

A bot's name tells you who fetched a page, not every use the content is put to. OpenAI says that if a site allows both GPTBot and OAI-SearchBot, it "may use the results from just one crawl for both use cases".

And a user agent is only a label, which any client can send. Ahrefs says its Bot Analytics now "verifies bots instead of trusting their User-Agent string", and Profound says Agent Analytics tells real AI bots from spoofed ones. OpenAI, Anthropic and Perplexity publish the IP addresses their bots use, so a request can be checked against them.

How is crawler traffic collected?

Crawler traffic has to be collected where the request arrives. A bot's request reaches your web server or CDN like any other, so the record comes from access logs, from CDN logs, or from code that runs at the edge before the page is served.

A browser tag alone cannot do it. Crawlers do not run JavaScript, so a tag that fires when a page loads in a browser records the people who arrive from an AI answer and none of the bots.

That is why the platforms that sell crawler data describe server-side collection. Profound says its Agent Analytics uses server logs instead of JavaScript trackers. Peec AI's Crawl Insights reads from your server logs, and so does Otterly.AI's Agent Analytics. Ahrefs' Bot Analytics captures bot activity through Cloudflare, Vercel, CloudFront or Fastly, with "No JavaScript required", and Conductor reads server log files.

Answyn's LLM Analytics works the same way. It installs in a Next.js site on Vercel, as a Cloudflare Worker, or through scheduled uploads of server log lines, and records which bots requested which paths, with live fetches kept apart from training crawls. An optional browser tag adds referral clicks from cited links; on its own, it would miss every crawler.

Is crawler tracking standard or an add-on?

Where AEO platforms offer it, crawler tracking usually comes inside a plan, with usage limits, rather than as a separately priced add-on. Which plan varies, and Semrush's Bot Analytics and Conductor's AI Crawler Activity sit in separate enterprise products.

From each vendor's own pages, read 4 November 2026
PlatformWhat it offersWhich plansExtra cost
Peec AICrawl Insights, from server logsEvery plan, up to 4.5M, 13.5M or 31.5M bot visits a month, or customIncluded
Otterly.AIAgent Analytics, from server logs, split into on-demand fetchers, search index and trainingStandard ($189 a month), Premium and Custom; not LiteIncluded, up to 200k events a month on Standard and 1M on Premium
ProfoundAgent Analytics, from server logs, telling real AI bots from spoofed onesTrial and Enterprise, with unlimited domains on bothIncluded; Enterprise price not published
AhrefsBot Analytics, through Cloudflare, Vercel, CloudFront or Fastly, verifying botsAll paid Ahrefs plans; Lite is £99 a monthFree during beta; later price not published
SearchableAgent AnalyticsEvery plan; Pro is $125 a month, or $100 billed annuallyIncluded; collection method not published
ScrunchAgent traffic monitoring, with a real-time feed of AI botsNot statedListed as included
SemrushA toolkit audit of whether AI bots can crawl you; Bot Analytics in Enterprise Site IntelligenceBot Analytics on Enterprise; Log File Analyzer plan not statedEnterprise price not published
ConductorAI Crawler Activity, from server log filesConductor Monitoring Enterprise packagesPrices not published
HubSpot AEONone on the pages we readNot applicableNot applicable
SE VisibleNone on the pages we readNot applicableNot applicable
Similarweb AI Search IntelligenceNone on the page we read; it reports AI referral traffic insteadNot applicableNot applicable
AnswynLLM Analytics: bots by path, live fetches against training crawls, and referral clicksNot publishedAsk when you enquire

Semrush, which Google's AI Overviews named in most answers to this question, puts an access check in its AI Visibility Toolkit: a site audit for issues "that might prevent AI bots from crawling your content", rather than a record of visits. Server-side Bot Analytics, covering 30 bots, is part of Enterprise Site Intelligence, and the Log File Analyzer reads uploaded log files, on a plan the page does not state.

Read the limits as well as the tick. Peec AI caps bot visits by plan and Otterly.AI caps events, and a feature that starts on a higher tier is an upgrade, whatever the pricing page calls it. Our guide to AEO software for UK marketing teams compares the same platforms on prompts, engines and price.

When is crawler tracking worth paying for?

Crawler data earns its cost when it would change what you do next. If you suspect a bot is blocked, a log shows which bots reach your pages and what status code each one gets. If the pages you most want cited never see a search crawler or a live fetch, the problem may be discovery rather than the copy. During a migration or redesign, when URLs, redirects and CDN rules change at once, a log shows whether the bots followed you to the new addresses.

It is the wrong first purchase if you do not yet know whether the answers name or cite you, because crawl data cannot tell you that. With one budget line, track the questions your buyers ask first; crawl data then helps explain the pages that are read and still not cited.

Check access for free before paying for anything. Our AI crawler access check reads your robots.txt for each named AI agent and requests your homepage as that agent, so it also catches a firewall or CDN that blocks what robots.txt allows. It is a snapshot of one page on one day, and it needs no account.

What should you ask a vendor before you sign up?

Ask these before you sign:

  1. Which bots does it recognise, and does it separate training crawls, search crawls and live fetches?
  2. How is it collected? Server logs, a CDN integration or edge code will see crawlers; a browser tag alone will not.
  3. Does it check a bot against the IP addresses its company publishes, or trust the user agent?
  4. Is it on the plan you are buying, and what are the monthly limits on events or bot visits?
  5. Can it put crawls and citations on the same row for each page, so you can find the pages that are crawled and never cited?

If you want crawls and citations side by side for every page on your site, enquire about Answyn.

Questions people ask

Is AI crawler tracking the same as citation tracking?
No. Crawler tracking records which AI bots fetched your pages. Citation tracking records which pages the answers linked to. A page can be crawled every week and never cited, so the two work best side by side.
Can a browser analytics tag show AI crawlers?
Not on its own. Crawlers do not run JavaScript, so a tag that fires in the browser never sees them, though it can still record people who click through from an AI answer. To see crawlers, collect from server logs, your CDN or code that runs at the edge.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot crawls content that may be used to train OpenAI's models. OAI-SearchBot crawls for ChatGPT's search features, and sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. ChatGPT-User visits a page when a user's question calls for it, and OpenAI says robots.txt rules may not apply to it. OpenAI treats each robots.txt setting independently.
Does blocking Google-Extended keep my site out of AI Overviews?
Not according to Google. It says Google-Extended does not affect inclusion in Google Search, and describes it as controlling whether content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. To limit what Search shows from a page, Google points to nosnippet, data-nosnippet, max-snippet and noindex.
Do AEO platforms include AI crawler tracking as standard?
Where they offer it, usually yes, inside a plan and with limits. On 4 November 2026, Peec AI included it on every plan, Otterly.AI from Standard, Profound on both its plans and Ahrefs free in beta on paid plans. Semrush's Bot Analytics and Conductor's AI Crawler Activity sat in enterprise products, and HubSpot AEO and SE Visible did not describe it at all.
How can I see which AI bots visit my site without a platform?
Your server or CDN access logs usually record each request's user agent, so filtering them for names such as GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot shows the visits. A platform adds sorting by purpose, checks against spoofed bots and the link to citations. Our free AI crawler access check tests whether named AI bots can reach your homepage at all.

Sources

Written by Answyn, which sells one of the platforms in the table below. Crawler documentation and vendor pages were read on 4 November 2026 and are re-checked quarterly. If something has moved, tell us at team@answyn.com and we will correct it and date the correction.

Run the numbers on your own brand.

Answyn puts the questions your buyers actually ask to every major engine, daily, and records who gets named and who gets cited.