Skip to content
growth auditor
Book

Agent Readiness scan

Visibility / Level 0 / live

Checks whether AI crawlers and answer engines can read, parse and cite your site: robots.txt rules, llms.txt, schema, canonicals and page basics.

Public footprint only. No login. About 20 seconds.

The 8 checks

  1. 01robots.txt reachablepasswarnfail
  2. 02AI crawlers allowed (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended)passwarnfail
  3. 03llms.txt presentpasswarnfail
  4. 04Homepage server-renderedpasswarnfail
  5. 05JSON-LD present (Organization plus page type)passwarnfail
  6. 06Canonical tag self-referencingpasswarnfail
  7. 07Sitemap reachablepasswarnfail
  8. 08Meta robots indexablepasswarnfail

What this scan checks

AI answer engines read the open web in two ways. Crawlers such as GPTBot, ClaudeBot and PerplexityBot fetch pages for training and retrieval. Live agents fetch a page at answer time. Both obey robots.txt, both prefer clean HTML with structured data, and both skip sites that make them work.

This scan fetches three things from your domain: robots.txt, llms.txt and the homepage HTML. It runs eight checks and returns each as a finding with evidence and a fix. One of them, server rendering, is reported as information until the rendering step ships.

Why it matters

In our first 40 test scans, 23 sites blocked at least one major AI crawler by accident, usually through a blanket rule copied from a template. 31 had no llms.txt. 12 had no JSON-LD at all. None of these sites appeared in AI answers for their own category terms.

How we score it

Each check is pass, warn or fail. Fails on crawler access and canonical hygiene weigh most. The module score feeds the Visibility pillar.

What to do with the result

Most fixes take under an hour. A blocked crawler is one line in robots.txt. A missing llms.txt is a text file. Schema and canonicals are usually a theme or CMS setting. If the fixes need content and entity work, the AI Search Visibility audit covers it.

Questions

What does the Agent Readiness scan check?
8 checks: robots.txt reachable; AI crawlers allowed (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended); llms.txt present; Homepage server-rendered; and more. Each returns Pass, Warn or Fail.
Does it need account access?
No. It runs on your public footprint from the domain alone.
Which audit fixes these findings?
The AI Search Visibility Audit ($1,500, 2 weeks).