How to Read the AI Crawler Log and Identify Bots - AEO God Mode

How to Read the AI Crawler Log and Identify Bots

What is the AI Crawler Log?

Unlike standard analytics tools (like Google Analytics) which track human visitors using JavaScript, the AEO God Mode Crawler Log tracks server-side bot visits. This is crucial because AI bots like ChatGPT-User or ClaudeBot do not execute JavaScript when reading your pages. The Crawler Log provides exact proof that your content has been digested by an AI model.

You can access the log by navigating to AEO God Mode > Crawler Log.

Understanding the Overview Cards

At the top of the dashboard, you will find four primary statistics:

  • 7-Day Crawls: The total number of successful visits by AI bots in the last week, along with the percentage trend compared to the previous week.
  • 30-Day Total: The total volume of crawls over the last month and the number of unique AI bots identified (e.g., ChatGPT-User, PerplexityBot, ClaudeBot).
  • Most Crawled Page: The specific URL on your site that AI models are requesting most frequently.
  • Content Blind Spots: The number of published posts on your site that have never been visited by any AI bot. These pages may need better internal linking or submission via your llms.txt file.

The Intelligence Panel

Below the overview, the dashboard features a tabbed intelligence panel.

Bot Activity

This tab displays individual cards for each registered AI crawler found on your site.

  • Visits: Total times the specific bot has visited.
  • Last seen: How recently the bot was detected.
  • Top page: The URL this specific bot focuses on most.

Occasionally, you may see a warning badge stating “Only checks robots.txt, doesn’t read content”. This indicates the bot is respecting protocols but isn’t actively compiling your articles for its language model. If a bot only ever checks your robots.txt without reading articles, it means your content hasn’t been triggered by a user prompt in that specific AI platform yet.

Top Crawled Pages (Pro)

This view lists your content ranked by crawl volume. This helps you identify which of your pages the AI platforms consider most valuable or authoritative. Pages with high crawl rates are highly likely to be used in AI Overviews or chat responses.

Blind Spots (Pro)

This tab isolates exactly which of your published posts are being ignored. If you have “Blind Spots,” it usually means the AI bots cannot easily navigate to those pages. To fix this, ensure the articles are linked from your homepage, category pages, or listed in your llms.txt file.

The Raw Log Table

At the bottom of the dashboard is the raw, paginated log of every single request. Use this table to:

  • Verify bot behavior: Confirm exactly which bot requested which URL and at what exact time.
  • Check HTTP status codes: A status of 200 means the bot successfully read the page. A 403 or 404 indicates an error or block that you need to resolve.
  • Identify aggressive crawling: If you see hundreds of requests per minute from the same bot, you can use the AI Crawler Allowlist module to block it if it begins impacting your server performance.