Skip to content
Cloudflare Docs

Analyze AI traffic

AI Crawl Control metrics provide you with insight on how AI crawlers are interacting with your website (Cloudflare zone).

To view AI Crawl Control metrics:

  1. Log in to the Cloudflare dashboard ↗, and select your account and domain.

  2. Go to AI Crawl Control.

    Go to AI Crawl Control

You can find meaningful information across both Crawlers and Metrics tabs.

View AI crawler activity

The Crawlers tab provides you with the following information:

MetricDescription
Total requestsTotal number of requests to crawl your website from common AI crawlers.
Requests by crawlerNumber of requests made by each AI crawler.

View AI Crawl Control metrics

The Metrics tab provides you with the following metrics to help you understand how AI crawlers are interacting with your website.

Analyze referrer data

Identify traffic sources with referrer analytics to understand discovery patterns and content popularity from AI operators.

  • View top referrers driving traffic to your site.
  • Understand discovery patterns and content popularity from AI operators.

Track crawler requests over time

Visualize crawler activity patterns over time using the Requests over time chart. You can group data by different dimensions to get more specific insights:

DimensionDescription
CrawlerTrack activity from individual AI crawlers (like GPTBot, ClaudeBot, and Bytespider).
CategoryAnalyze crawlers by their purpose or type.
OperatorDiscover which companies (such as OpenAI, Anthropic, and ByteDance) are crawling your site.
HostBreak down activity across multiple subdomains.
Status CodeMonitor HTTP response codes (200s, 300s, 400s, 500s) to crawlers.

Understand what content is crawled

The Most popular paths table shows you which pages on your site are most frequently requested by AI crawlers. This can help you understand what content is most popular with different AI models.

ColumnDescription
PathThe path of the page on your website that was requested.
HostnameThe hostname of the requested page.
CrawlerThe name of the AI crawler that made the request.
OperatorThe company that operates the AI crawler.
Allowed requestsThe number of times the path was successfully requested by the crawler.

You can also filter the results by path or content type to narrow down your analysis.

Filter and export data

You can use the date filter to choose the period of time you wish to analyze. To export your data, select Download CSV. The downloaded file will include all applied filters and groupings.

Filter options:

  • Past 24 hours

The values of the AI Crawl Control metrics will update according to your filter.