Agentic SEO has two meanings. The first is making a site easy for AI agents to find, read and act on: crawler rules, clean HTML, structured data and forms an agent can complete. The second is using AI agents to do SEO work. This page covers the first. We call it agent readiness, because the work goes beyond search.

Should I block GPTBot?

Only if you do not want your content used for model training. OpenAI says GPTBot crawls content that may be used to train its foundation models. Blocking it does not block OAI-SearchBot, which OpenAI uses to surface sites in ChatGPT search. Many sites block GPTBot and allow OAI-SearchBot. Make the choice per bot, on purpose, and check that no blanket rule overrides it.

Do AI agents respect robots.txt?

Crawlers mostly do. Fetchers triggered by a user often do not. Anthropic says its bots honor robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the action. Perplexity says Perplexity-User generally ignores robots.txt. robots.txt controls automated crawling well and controls user-triggered fetches poorly.

WebMCP is a proposed web standard from Chrome that lets a site expose structured tools for AI agents. You can define tools in JavaScript, or annotate existing HTML forms so an agent can call them directly instead of guessing where to click. Chrome opened an origin trial from Chrome 149. It is early. Watch it before you build on it.

Does agent readiness affect Google rankings?

Mostly indirectly. Google says you need no special files or markup to appear in its AI features, and it ignores llms.txt. The parts of agent readiness that overlap with technical SEO, such as canonical tags, one H1 per page and crawlable HTML, help Google too. The AI-specific parts mainly affect ChatGPT, Claude and Perplexity.

Library / Visibility / Oct 28, 2026

Agent Readiness: Is Your Site Ready for AI Agents?

Agent readiness is how well AI crawlers and AI agents can reach, read and act on your website. It covers robots.txt rules for AI bots, content in plain HTML, structured data, llms.txt and forms an agent can complete. Some teams call this agentic SEO. A site that blocks or confuses agents drops out of AI answers and agent tasks.

Definition: Agent readiness is the degree to which software acting for a user can find, parse and complete tasks on your site without a human stepping in.

What is agentic SEO?

The term has two meanings. One is optimizing a site for AI agents that browse on a user’s behalf. The other is using AI agents to do SEO work. This page covers the first. We prefer “agent readiness” because the work reaches past search: an agent may compare your pricing, fill in your demo form or check out.

Why does agent traffic matter now?

Cloudflare reported that AI “user action” crawling increased by over 15x in 2025. These are fetches made because a person asked an assistant something.
In the same review, AI bots other than Googlebot made up 4.2% of HTML request traffic on Cloudflare’s network. Googlebot alone was 4.5%.
On July 1, 2025, Cloudflare began blocking AI crawlers by default and asking every new domain at sign-up whether to allow them.

That last change matters when you check your own site. A setting chosen in a sign-up flow, or a robots.txt line copied from a template, can block AI search crawlers without anyone on the growth team knowing.

Which AI bots visit your site?

There are three kinds, and each needs its own decision.

Type	Examples	What it does	Follows robots.txt?	If you block it
Training crawler	GPTBot, ClaudeBot	Collects pages that may train future models	Yes, per OpenAI and Anthropic	Future content is left out of training
Search crawler	OAI-SearchBot, Claude-SearchBot, PerplexityBot	Indexes pages so the engine can cite them in answers	Yes	You drop out of that engine’s answers
User-triggered fetcher	ChatGPT-User, Claude-User, Perplexity-User	Fetches a page because a user asked	Often not	Partial at best

Sources: OpenAI, Anthropic, Perplexity. OpenAI says robots.txt rules “may not apply” to ChatGPT-User. Perplexity says Perplexity-User “generally ignores robots.txt rules”. Anthropic says its bots honor robots.txt and that blocking Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results”. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that changes take about 24 hours to apply.

The common pattern we recommend: allow search crawlers, decide on training crawlers as a business choice, and write the reason next to each rule.

What makes a site agent-ready?

Deliberate robots.txt. One named rule per AI bot. No blanket disallow that catches search crawlers by accident. (30 min)
Content in the HTML. Prices, plans and product facts present in the page source, not only after scripts run. We check what the raw HTML contains. (1 week on a script-heavy stack)
Structured data. JSON-LD for Organization, Product or SoftwareApplication, and Article on reference pages. It tells an agent what each page is. (2 hours)
Canonical hygiene. Each page declares one self-referencing canonical, so an agent does not quote a duplicate. (2 hours)
A reading guide. An llms.txt file and Markdown versions of key pages. Low cost, uneven uptake. (30 min)
Pages and forms an agent can operate. Google says browser agents may analyze screenshots, inspect the DOM and interpret the accessibility tree (Google Search Central). Labeled fields, real buttons and clear error text serve both agents and screen readers. (1 day)
A plan for actions. Chrome’s WebMCP is “a proposed web standard to help you build and expose structured tools for AI agents”, in origin trial from Chrome 149. For commerce, Stripe and OpenAI released the Agentic Commerce Protocol on September 29, 2025. Google’s guide also names the Universal Commerce Protocol as emerging. None of these is required today. They show where agent traffic is heading.

Agent readiness is the access layer of generative engine optimization. Without it, no amount of content work gets you cited.

What we check in the Growth Scan

The Agent Readiness module fetches robots.txt, llms.txt and your homepage HTML, then runs nine checks.

Check	Pass	Warn	Fail
robots.txt exists and parses	Present, valid, returns 200	Present with syntax errors	Missing, or returns 5xx
AI crawler rules (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot)	Each bot has a deliberate rule; search crawlers allowed	Training crawlers blocked by a blanket rule	Search crawlers blocked, or whole site disallowed
llms.txt present and non-empty	Present, has an H1, links resolve	Missing, or present with broken links or no H1	n/a
JSON-LD present	Organization plus a page-type schema	Organization only	None
Canonical tag	Present and self-referencing	Points to another URL	Missing
Viewport meta	Present	n/a	Missing
Title length	30 to 60 characters	61 to 70, or under 30	Missing or over 70
Meta description length	70 to 160 characters	Outside the range	Missing
Exactly one H1	One	n/a	Zero, or more than one

A missing llms.txt is a Warn, never a Fail. The evidence that engines read it is thin, and we do not score what we cannot show matters.

Run a free Growth Scan on your site to see where you stand.

Sources

Questions

What is agentic SEO?: Agentic SEO has two meanings. The first is making a site easy for AI agents to find, read and act on: crawler rules, clean HTML, structured data and forms an agent can complete. The second is using AI agents to do SEO work. This page covers the first. We call it agent readiness, because the work goes beyond search.
Should I block GPTBot?: Only if you do not want your content used for model training. OpenAI says GPTBot crawls content that may be used to train its foundation models. Blocking it does not block OAI-SearchBot, which OpenAI uses to surface sites in ChatGPT search. Many sites block GPTBot and allow OAI-SearchBot. Make the choice per bot, on purpose, and check that no blanket rule overrides it.
Do AI agents respect robots.txt?: Crawlers mostly do. Fetchers triggered by a user often do not. Anthropic says its bots honor robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the action. Perplexity says Perplexity-User generally ignores robots.txt. robots.txt controls automated crawling well and controls user-triggered fetches poorly.
What is WebMCP?: WebMCP is a proposed web standard from Chrome that lets a site expose structured tools for AI agents. You can define tools in JavaScript, or annotate existing HTML forms so an agent can call them directly instead of guessing where to click. Chrome opened an origin trial from Chrome 149. It is early. Watch it before you build on it.
Does agent readiness affect Google rankings?: Mostly indirectly. Google says you need no special files or markup to appear in its AI features, and it ignores llms.txt. The parts of agent readiness that overlap with technical SEO, such as canonical tags, one H1 per page and crawlable HTML, help Google too. The AI-specific parts mainly affect ChatGPT, Claude and Perplexity.