How to Fix Your robots.txt for AI Crawlers (5-Minute Guide)

Here is a fact that surprises most website owners: your robots.txt file is very likely blocking one or more major AI crawlers right now, even if you never intended to.

Our research on AI crawler blocking found that over 40% of the top 100K websites block at least one major AI crawler. The most common cause is not deliberate blocking — it is overly restrictive default rules.

Many robots.txt files include a blanket User-agent: * rule with restrictive Disallow patterns. Others use CDN or security features that aggressively block unknown bots. Some CMS platforms ship with restrictive defaults that were written before AI crawlers existed.

The result: your content is invisible to ChatGPT, Claude, Perplexity, and other AI systems. You are missing citations, traffic, and the compound visibility benefits that come from being part of AI-generated responses.

The fix takes under 5 minutes. This guide walks through exactly what to do, with copy-paste templates and platform-specific instructions.

Each major AI platform operates its own web crawler. These bots identify themselves with specific user agent strings that you can explicitly allow or block in your robots.txt. Here are the five that matter most in 2026:

GPTBot is OpenAI’s crawler, used by ChatGPT, ChatGPT Plus, and the ChatGPT API. It is the highest-volume AI crawler and the one most commonly blocked. Blocking GPTBot means your content cannot appear in ChatGPT responses.

ClaudeBot is Anthropic’s crawler, used by Claude and Claude-based applications. It follows robots.txt directives strictly and is considered one of the most respectful AI crawlers in terms of request frequency.

PerplexityBot powers Perplexity AI, the AI-first search engine. Perplexity explicitly cites its sources, so being accessible to PerplexityBot directly translates to visible, linked citations.

Google-Extended is Google’s AI-specific crawler, separate from Googlebot. It feeds content to Google’s Gemini and AI Overviews. Importantly, blocking Google-Extended does not affect your regular Google Search rankings — it only affects whether your content appears in AI-generated responses.

CCBot is the Common Crawl bot. Common Crawl’s open dataset is used as training data by many AI systems. Allowing CCBot ensures your content is represented in these foundational datasets.

A robots.txt file is a plain text file at your domain root (yoursite.com/robots.txt) that tells web crawlers which pages they can and cannot access. It uses a simple directive syntax: User-agent identifies the bot, Allow grants access, and Disallow restricts it.

The diagram below shows the anatomy of an AI-friendly robots.txt file. The key principle is explicit allows for AI crawlers combined with sensible disallows for sensitive paths. Do not rely on a blanket User-agent: * rule for AI bots — always specify them individually.

Here is a complete, production-ready robots.txt template. Copy this, customize the disallow paths for your site, and deploy. This template explicitly allows all five major AI crawlers while maintaining sensible restrictions on sensitive paths.

The template is organized in three sections: global rules that apply to all bots, explicit AI crawler rules, and metadata (sitemap location). Adjust the Disallow paths under User-agent: * to match your site’s actual directory structure.

The process for editing robots.txt varies by platform. Here are step-by-step instructions for the most common setups.

WordPress: Navigate to your site root via FTP or file manager. The robots.txt file is in the root directory (same level as wp-config.php). If it does not exist, create it. If you use Yoast SEO, go to Yoast → Tools → File Editor → robots.txt. If you use Rank Math, go to Rank Math → General Settings → Edit robots.txt. After editing, clear any page cache.

Shopify: Shopify auto-generates robots.txt and does not allow direct editing via the admin panel. To customize it, go to Online Store → Themes → Edit Code → Add a new template called robots.txt.liquid. Shopify’s documentation provides the syntax for adding custom rules. This override approach lets you add AI crawler allows while keeping Shopify’s default rules.

Vercel / Next.js: Place a robots.txt file in your public/ directory. For dynamic generation, create an API route at app/robots.txt/route.ts that returns the content with text/plain content type.

Cloudflare Pages: Place the file in your project’s build output root. Alternatively, use a Cloudflare Worker to dynamically serve robots.txt content.

Apache (.htaccess): If you need to redirect or override robots.txt, add a rewrite rule in your .htaccess file. Ensure the rule serves the correct file with text/plain content type.

Nginx: Serve robots.txt with a location block in your nginx configuration that points to the file path and sets the correct content type.

After deploying your updated robots.txt, verify it in three ways:

Step 1: Direct access. Navigate to yoursite.com/robots.txt in your browser. Confirm the AI crawler rules appear as expected. Check for syntax errors — extra spaces, missing colons, or incorrect indentation can cause parsing failures.

Step 2: Header verification. Use curl -I yoursite.com/robots.txt to check the response headers. The status code should be 200 and the content type should be text/plain. A 301 redirect to a different file or a 403 forbidden response means something is blocking access.

Step 3: Robots.txt tester. Use Google Search Console’s robots.txt Tester (under the legacy tools) to check whether specific user agents are blocked or allowed for specific paths. Test each AI crawler user agent manually.

Step 4: CDN and firewall check. If you use Cloudflare, check your WAF and Bot Fight Mode settings. Some CDN configurations aggressively challenge or block bots regardless of robots.txt. Cloudflare’s Bot Fight Mode in particular is known to block legitimate AI crawlers. You may need to create a WAF rule that allows specific user agents.

Monitor your server logs over the following 7–14 days to confirm AI crawlers are actually visiting. If you allow GPTBot but never see it in your logs, there may be a firewall or CDN rule intercepting requests before they reach your server.

robots.txt is necessary but not sufficient. Even if your robots.txt explicitly allows AI crawlers, other layers of your infrastructure might block them.

Cloudflare Bot Fight Mode is the most common offender. When enabled, it challenges or blocks bots that are not on Cloudflare’s verified bot list. As of early 2026, not all AI crawlers are on this list. Check your Cloudflare dashboard under Security → Bots and ensure your Bot Fight Mode settings are not interfering. Consider adding a custom WAF rule that skips the challenge for known AI user agents.

AWS WAF and CloudFront can similarly block bots based on user agent patterns. If you use AWS, check your WAF rules for any broad bot-blocking patterns.

Rate limiting is another consideration. AI crawlers are generally well-behaved, but if your rate limiting is too aggressive (e.g., 10 requests per minute per IP), you may inadvertently throttle them. Check your CDN and server rate limiting configuration.

Rendering gates are the final barrier. If your content is behind a JavaScript rendering wall and you do not have server-side rendering, AI crawlers (which typically do not execute JavaScript) will see an empty or skeleton page. Use server-side rendering or static generation for all content pages. Our complete AI readiness guide covers this in depth under the page speed factor.

After fixing robots.txt, run a free AgentReady scan to check for these additional access barriers. The Bot Access factor evaluates all layers, not just robots.txt.