What is robots.txt?
TL;DR
robots.txt is a plain-text file at your site's root that tells web crawlers (including AI bots) which parts of your site they are allowed to access.
Last updated: 2026-03-09
Definition#
robots.txt is one of the oldest standards on the web. It is a plain-text file placed at
https://yoursite.com/robots.txt that provides instructions to web crawlers about which pages or sections of your site they may or may not access.
The file uses simple rules. User-agent specifies which bot the rule applies to. Allow and Disallow specify which paths the bot can or cannot visit. Sitemap points to your XML sitemap for easier discovery.
In the context of AI readiness, robots.txt has taken on new importance. AI companies have introduced their own crawlers (GPTBot, ClaudeBot, PerplexityBot), and your robots.txt rules determine whether these bots can access your content. Blocking them means your content will not appear in their AI platforms.Why It Matters for AI Readiness#
Your robots.txt file is the gateway to AI visibility. If AI crawlers are blocked, nothing else matters — your content is invisible to AI models. The AgentReady™ scanner checks your robots.txt as the first step in evaluating Bot Access.
See our guide on fixing robots.txt for AI crawlers for a complete template and step-by-step instructions.
Related Concepts#
robots.txt works with meta robots tags (page-level directives), sitemaps (page discovery), and AI crawlers (the bots it controls).
Related Pages
Was this page helpful?