Skip to content

AI SEO Audit With Claude Code: A Real Site Walkthrough

An AI SEO audit asks one question a normal audit doesn't: can ChatGPT, Perplexity and Claude fetch, read and identify you? I ran it on my own site with Claude Code. Here are the requests, the evidence, the one real problem it found, and the skill.

"Can AI crawlers read my site? AI SEO audit" beside a Claude Code terminal: one saeedseo.com post requested as a browser, GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot and Bingbot, every one answered 200.
Table of Contents9 sections

Get my next blog post first

Real tests on Claude and SEO tools, with the numbers and what failed. About once a week.

Unsubscribe anytime. Privacy

On 4 November I gave Claude Code one job: check whether ChatGPT, Perplexity, Claude and Google can actually read my site. Every check had to leave its raw evidence in a file.

Fourteen user agents and sixteen URLs later, the site mostly passed. But all six of my published posts had the same small fault, and I'd never noticed it.

The short version: an AI SEO audit is your technical audit plus a few checks on access, snippets, reading cost and identity.

Each check proves less than it looks. So the evidence and its limits sit next to every finding.

Just want the steps? Jump to the checklist, or grab the skill file.

Key Takeaways

  • Access, reading and being cited are three questions. This audit covers two.
  • A 200 for a fake "GPTBot" request proves how your server answers a header. It doesn't prove the real crawler gets in, and it says nothing about whether you'll be cited.
  • robots.txt is only half of access. My CDN blocked a common user agent on every path I tested, robots.txt included.
  • Google needs no special files or schema.
  • The real find: a newsletter heading opened the outline of every post, before the article's first section.

What an AI SEO audit checks

A normal SEO audit asks whether Google can crawl and index you. The AI layer adds other crawlers, with their own names and their own rules, that you can block without touching Google.

QuestionWhat answers itIn this audit?
Can the assistants fetch the page?Requests per user agent, robots.txt, the CDN's bot rulesYes, except the CDN dashboard
Can they read it, and cheaply?Snippet controls, the heading outline, a markdown copyYes
Do they know who wrote it?Structured data, author linksYes
Do they cite it?Real questions, asked many timesNo: that's a separate test

Google's part is the simplest. Its guide to AI features says a page has to be indexed and eligible for a snippet. Nothing extra.

My setup, and what it can't show

Sitesaeedseo.com, a static Next.js site on a Cloudflare Worker
When4 November 2026, around midnight Bali time
PagesAll 16 URLs in the sitemap; the home page and one post for the crawler test
ToolClaude Code sending plain HTTP requests, so you can repeat each one with curl

What it can't show: whether the real crawlers visit.

OpenAI publishes its crawlers' IP ranges, so real visits can be checked in server logs or the CDN dashboard. Anthropic doesn't publish its ranges. This run didn't check logs.

The audit, in order

1. Request the same page as each crawler

Ask for the page with each crawler's user-agent string. A different status, or a different body, means something treats that bot differently.

Terminal output: one saeedseo.com blog post requested as a browser, GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot and Bingbot. Every row returns 200 and the same 78,265 bytes.

I sent each crawler's user agent, on the home page and one post.

Google-Extended isn't in the list on purpose. It's a robots.txt token for AI training, and it never visits.

2. Read robots.txt the way a crawler does

**A named group replaces the * group for that bot.**

Google's robots.txt spec says it plainly, and it catches a lot of sites, because most people expect the general rules to apply to every crawler and only add the exceptions.

Write User-agent: GPTBot with only Allow: / under it, and GPTBot ignores every Disallow you wrote for everyone else.

So the audit lists each group, and flags one that's missing lines the * group has.

3. Check snippet and index controls

No snippet, no AI Overview. The audit reads each page's robots meta tag, the X-Robots-Tag header, the canonical, and any nosnippet, max-snippet or data-nosnippet.

4. Ask for a markdown copy

Request the page with Accept: text/markdown. Some sites, mine included, answer with a plain markdown version.

Terminal output: requesting the GEO vs SEO post with Accept text/markdown returns HTTP 200 with content-type text/markdown, a canonical Link header and Vary: Accept. The markdown copy is 18,322 bytes; the HTML is 78,265.

Nobody has shown that an assistant prefers it. I list it as an observation, never as a requirement, and I'd be wary of any audit tool that scores you down for not having one.

5. List the heading outline

Pull every page's H1 and H2s in source order. Does anything that isn't the article come first?

6. Read the structured data

List the schema types per page. For the Person behind the site, check that every post's author points at the same @id. Here's how I set mine up, with the commits.

What I found on saeedseo.com

Here's the crawler test, on the home page and on a post:

User agentHome pageBlog post
Browser200200
GPTBot, OAI-SearchBot, ChatGPT-User200200
PerplexityBot, Perplexity-User200200
ClaudeBot, Claude-User, Claude-SearchBot200200
Googlebot, Bingbot, Applebot200200
CCBot, Bytespider200200

Boring? That's what you want here. Every agent got the same bytes as the browser (124 KB for the home page, 78 KB for the post), so nothing was cloaked or challenged.

Then a surprise from my own tooling. When Claude Code fetched the sitemap with Python's built-in library, Cloudflare sent back a 403.

Terminal output: three saeedseo.com paths, the home page, robots.txt and llms.txt, each requested twice. Python's default user agent gets 403 on all three; curl gets 200 on all three.

curl, python-requests and Go all got through. So did a request with no user agent at all.

The named AI crawlers aren't affected. But a home-made agent left on Python's default is locked out of my robots file.

The lock is Cloudflare's Browser Integrity Check, which I turned on for this site. The 403 says "Error code: 1010", the block it serves when a request's headers look like a script.

The other checks were clean:

CheckResult on all 16 URLs
Status200
Robots metaindex, follow, no X-Robots-Tag
CanonicalEqual to the URL
Snippet blocksNone
Markdown copyServed on every page, 3.5% to 28% of the HTML's size
robots.txt8 AI user agents named (six crawlers, plus the Google-Extended and Applebot-Extended training tokens), each group repeating the * group's three Disallow lines

The heading that opened every post (fixed)

This was the finding. All six posts published at the time had "Get my next blog post first" as their first H2, before "Key Takeaways".

It was the sidebar's newsletter box. A reader never saw the problem.

In the source, a sign-up heading came before the article's first section. Nothing shows a model was confused by it.

It was still the wrong first heading. At the end, a second newsletter title was an H2 too.

Terminal output: the first four H2 headings in the GEO vs SEO post's source. On the live site before the fix, the first is "Get my next blog post first", marked red, before Key Takeaways. On the live site after the fix, the first is Key Takeaways, marked green.

The fix was small. The newsletter and dialog titles became paragraphs, with the same font and size, still named for screen readers.

"Related Posts" stayed a heading. It really is a section.

Fixed and live on 5 November 2026. Every post's first H2 is now "Key Takeaways".

Observations that need no fix

  • The Person schema has an @id, nine profile links, and every post's author points at it.
  • There's an llms.txt. Google Search ignores it, so it's here as an observation.

The script and the Claude Code skill

The skill is the audit above, written down so it runs the same way every time. It takes a site, picks up to ten URLs from the sitemap, runs the six checks and writes a short report.

Three rules in it matter more than the checks:

  1. No evidence, no finding. Every line points at a file and a row.
  2. Label everything as a finding, an observation or a limit.
  3. Never edit the site. The audit reports; I decide what changes.

Get the AI SEO audit skill

The complete ai-seo-audit skill, ready to save as .claude/skills/ai-seo-audit/SKILL.md. Free, straight to your inbox.

How I'd verify a fix

  1. Deploy the change.
  2. Re-run the same check on the live URL, with the same command.
  3. Put the date and the new output next to the old one.

Don't stop at "it works locally." My outline fix passed on a dev server first, and only counted as fixed once the live HTML said so.

Getting crawled is only the first step to a ChatGPT citation. The rest is tracing which pages it pulls and cites.

AI SEO audit checklist

CheckPasses when
Crawler user agentsEach gets the same status and body as a browser
CDN bot rulesVerified bots allowed, checked in the dashboard
robots.txt groupsNo named group drops the * group's Disallow lines by accident
Snippet controlsNo noindex, nosnippet or max-snippet:0 where you want to be quoted
Heading outlineThe first H2 in the source belongs to the article
IdentityOne Person or Organization @id, used by every author line
CitationsTested separately, with real questions asked many times

Common Questions

What is an AI SEO audit?

A technical SEO audit with checks for AI assistants added: whether ChatGPT, Perplexity and Claude crawlers can fetch the pages, whether snippets are allowed, how the page outline reads, and whether structured data identifies the author.

Does a 200 response to a GPTBot user agent mean OpenAI can crawl my site?

No, that alone proves little. A request with a copied user-agent string only shows how the server answers that header. OpenAI says to verify its crawlers by their published IP ranges, so check server logs or the CDN's verified-bot settings too.

Do I need llms.txt or special schema for Google's AI Overviews?

No. Google's guide for its AI features says a page must be indexed and eligible for a snippet in Google Search, and that no special files or extra structured data are needed.

Can a robots.txt rule for one AI crawler cancel my other rules?

Yes. Under Google's robots.txt spec, a group naming a specific crawler replaces the general * group for that crawler. Any Disallow lines you want it to follow must be repeated inside its own group.

Hiring for AI search? Let’s talk.

Email me the job description, or book thirty minutes now. You’ll hear back within one working day.

Put your email and confirm it from your inbox. The file comes straight after.

You'll also get new posts, about once a week. Unsubscribe anytime. Privacy