Fixing Crawlability for AI Engines

Last updated: June 24, 2026

Crawlability is the number one SEO gap. If your CDN, WAF and host ruleset, or robots.txt is silently blocking AI crawlers, nothing else you do matters because the engines never reach your content. The good news: it's usually a one-hour fix. This article shows you how to diagnose and fix it.

The crawlers to allow

These are the named user agents that matter for AI visibility right now:

Engine

User agent

Why it matters

OpenAI ChatGPT

GPTBot, ChatGPT-user

GPTBot builds OpenAI's search index; ChatGPT-user fetches live when someone asks

Common Crawl

CCBot

Training data that feeds many LLMs, including Claude and open-source models

Perplexity

PerplexityBot

Live retrieval for every Perplexity answer

Anthropic Claude

ClaudeBot (also anthropic-ai), Claude-Web

Live retrieval for Claude web search

Microsoft Bing / Copilot

Bingbot

Bing's index powers Copilot and is ChatGPT's fallback retrieval layer

Google

Google-Extended

Opts your content into Gemini and Google AI training; Google's index powers AI Overviews

Apple

Applebot-Extended

Controls whether your content is used for Apple Intelligence

Meta

Meta-ExternalFetcher

Live fetch for the Meta AI assistant

Add more as new engines emerge (Mistral, DeepSeek). When in doubt, allow the crawler. Most teams lose AI visibility by over-blocking, not by allowing the wrong bot.

Diagnose in three steps

Step 1: Test bot access. Use Rankscale's Page Audit V2 to check whether all bots are allowed. It tests the crawlers for each AI search engine at once.

Step 2: Check the three layers that can block a bot: CDN/provider, firewall, and host. A blocked crawler is stopped at one of three layers, and a pass at one doesn't clear the others. Especially now that providers are becoming more strict on bot requests (due to increased traffic), keep an eye on your inbox for any changes made by your providers. Work through them in the following order.

  • CDN / client provider (the network in front of your site): Cloudflare, Fastly, Vercel, or Akamai can block at the IP or network level. AI crawlers come from known cloud IP ranges, so blanket IP blocks, geo-fences, or a global "AI bots" toggle reject them before any other rule runs. If Cloudflare's "Block AI Scrapers and Crawlers" toggle is on, turn it off for the pages you want AI to cite, and leave it on only for paywalled or gated content.

  • Firewall (WAF): Cloudflare, Fastly, and AWS WAF ship with "AI Scraper" or "AI Bot" block rulesets that override robots.txt. Log into the console and search for rules targeting GPTBot, CCBot, or "AI scrapers." Disable them or add explicit allow rules for the user agents above.

  • Host (your origin server): server-level rules (.htaccess, Nginx config), aggressive rate limits, or IP throttling can still drop bots even when the CDN and WAF allow them. Whitelist the bot user agents and raise request thresholds for known crawlers.


Step 3: Read robots.txt. Open https://yourdomain.com/robots.txt and look for any Disallow: / rule targeting one of the four user agents. A common anti-pattern:

User-agent: GPTBot

Disallow: /

This blocks ChatGPT entirely. Remove the Disallow, or scope it narrowly (for example Disallow: /admin/).

The "not indexed on Google" or “not indexed on Bing” check

Google AI Overviews and AI Mode source directly from Google's index. If a page is crawled but not indexed, it won't appear regardless of quality. Check Google Search Console → URL Inspection. If the status is anything other than "URL is on Google," fix the indexing issue (canonical tags, noindex directives, duplicate content) before moving on. Do the same for Bing since Bing can acts as a fallback mechanism for AI Search Engines when Google becomes too expensive.

Common fix patterns

  • robots.txt too broad: replace a blanket Disallow: / with narrow paths (/admin/, /checkout/, /api/).

  • WAF catching legitimate bots: add explicit Allow rules for the four user agents.

  • Rate limits too aggressive: whitelist bot user agents or raise request thresholds for known crawlers.

  • IP-based blocking: AI crawlers come from cloud IP ranges, so blocking cloud IPs blocks them. Move to user-agent-based rules.

Do this now

Run Page Audit V2 to check whether every bot can reach your site. If any fail, fix robots.txt or your WAF this morning and retest within the hour.

Related Articles

  • What Is an SEO Gap?

  • Fixing JavaScript-Rendered Content

  • Adding JSON-LD Schema for AI Visibility

  • Fixing an Entity Gap