What is Meta Robots?
TL;DR
Meta robots is an HTML tag that gives page-level instructions to search engine and AI crawlers about whether to index, follow links, or cache a specific page.
Last updated: 2026-03-09
Definition#
The meta robots tag is an HTML element placed in the
<head> of a page that provides instructions to crawlers about that specific page. While robots.txt controls access at the directory level, meta robots controls behavior at the page level.
Common values include index (allow indexing), noindex (prevent indexing), follow (follow outbound links), nofollow (do not follow links), noarchive (do not cache the page), and nosnippet (do not show a text snippet in results).
For example, <meta name="robots" content="noindex, follow"> tells crawlers not to index the page but still follow its outbound links. You can also target specific bots with <meta name="googlebot" content="noindex">.Why It Matters for AI Readiness#
Incorrect meta robots tags can accidentally hide your important pages from AI crawlers. A
noindex tag on a key product page means AI models cannot find or cite it. The AgentReady™ scanner checks for meta robots issues as part of the Bot Access factor.
Review your meta robots tags to make sure important pages are set to index, follow.Related Concepts#
Meta robots works alongside robots.txt (site-level access control), canonical tags (preferred URL designation), and noindex (specific directive to prevent indexing).
Related Pages
Was this page helpful?