NLWeb, MCP, and llms.txt: The Three Protocols That Will Define the Agentic Web

For thirty years, the web had one consumer: the human browser. HTML was designed for human eyes. CSS for human aesthetics. JavaScript for human interaction. That era is ending.

AI agents are the web's second consumer. And they need their own stack. Not a replacement for HTML — an addition. A parallel layer that lets machines read, understand, and act on your content without scraping, guessing, or hallucinating.

AgentReady™ has tracked three protocols emerging as the foundation of this new stack: llms.txt, NLWeb, and MCP. Each serves a distinct purpose. Together, they form the complete communication layer between your website and the AI agents that will increasingly drive traffic, transactions, and trust.

Think of it like the original web stack. HTML gave structure. CSS gave presentation. JavaScript gave behavior. The agentic web has the same three-layer logic — just for a fundamentally different audience.

The simplest protocol is often the most powerful. llms.txt is a plain text file placed at your domain root that tells AI models what your site is, what it offers, and how to navigate its content. Think of it as robots.txt's smarter sibling.

Where robots.txt says "don't go here," llms.txt says "here's what matters." It provides structured context — your business description, key pages, content hierarchy, and preferred citation format. It takes 15 minutes to create and immediately improves how AI models understand your site.

Adoption is accelerating. As of early 2026, roughly 7.2% of the top 10,000 sites have implemented llms.txt, up from under 1% in mid-2025. The protocol was proposed by Jeremy Howard (fast.ai founder) and has gained traction because of its radical simplicity — no API, no server changes, just a text file.

If llms.txt is a welcome sign, NLWeb is a concierge desk. Developed by Microsoft and released as an open protocol in mid-2025, NLWeb lets AI agents ask your website questions in natural language and receive structured, authoritative answers.

The protocol works through a /.well-known/nlweb endpoint. An AI agent sends a query — "What sizes does the blue running shoe come in?" — and your NLWeb endpoint returns a structured response with the answer, confidence level, and source citation. No scraping. No inference. Direct, authoritative answers from the source.

This is a fundamental shift. Traditional search required AI to crawl your HTML, parse it, and hope it extracted the right information. NLWeb makes your site conversational. It can answer questions about products, services, policies, pricing — anything you configure.

Adoption is still early. Roughly 3% of commercial websites have NLWeb endpoints, concentrated in tech and SaaS. But Microsoft's backing and the protocol's alignment with how AI agents actually work suggest rapid growth through 2026-2027.

The Model Context Protocol, built by Anthropic, completes the stack. Where llms.txt lets AI read and NLWeb lets AI understand, MCP lets AI agents take action on your behalf — booking appointments, checking inventory, initiating purchases, managing accounts.

MCP provides a standardized interface for AI agents to interact with external systems. For website owners, this means exposing specific capabilities — your booking system, your inventory API, your customer service tools — through a protocol that any AI agent can use safely and predictably.

The key word is "safely." MCP includes built-in permission models, capability declarations, and transaction boundaries. An AI agent can't accidentally delete your database. It can only perform the actions you explicitly expose, with the constraints you define.

MCP adoption is concentrated in developer tools and platforms today, with major integrations across IDE tools, cloud platforms, and SaaS products. For most website owners, MCP is the third priority — you need llms.txt and NLWeb first. But for transactional sites (e-commerce, booking, SaaS), MCP readiness will become competitive advantage fast.

These three protocols aren't competing. They're complementary layers in a single stack. Each serves a distinct function, and implementing them in order creates a natural progression from basic AI visibility to full agentic integration.

The implementation order matters. Start with llms.txt (hours, not days). Add NLWeb when you're ready for conversational AI interaction. Implement MCP when you need AI agents to take transactional actions on your site. Most sites in 2026 should focus on layers one and two.

Let's be honest about where things stand. These protocols are early. But "early" is exactly where the opportunity lives.

llms.txt has the broadest adoption at roughly 7% of major sites, driven by its zero-cost implementation. NLWeb sits at approximately 3%, growing among tech-forward companies. MCP adoption among public websites is harder to measure since it's typically implemented as API integrations rather than public endpoints, but major platforms including Slack, GitHub, and cloud providers have launched MCP servers.

The pattern mirrors early HTTPS adoption. In 2014, fewer than 30% of sites used HTTPS. Google's ranking signal announcement triggered a wave, and within four years it was essentially universal. AI protocols are following the same curve, just faster. The trigger events — AI search reaching mainstream adoption, Google potentially incorporating protocol signals — could compress the timeline dramatically.

The answer is almost always llms.txt first. It's the fastest to implement, requires no infrastructure changes, and provides immediate value. You can create and deploy a llms.txt file in 15 minutes with our guide.

NLWeb should be your second priority if you have content that AI agents would want to query — product catalogs, service descriptions, documentation, FAQs. The implementation requires a backend endpoint but the payoff is substantial: instead of AI guessing about your content, it asks you directly.

MCP is your third priority and is most relevant if you operate a transactional platform. If customers book, buy, or subscribe through your site, MCP readiness will matter. But get the foundation right first.

The sites that implement all three layers will be the ones AI agents trust, recommend, and transact through. That's not a prediction — it's the architecture these protocols are explicitly designed to enable. Check your current protocol readiness and start building your agentic web presence today.