---
title: "AI SEO Audit With Claude Code: A Real Site Walkthrough"
description: "I ran an AI SEO audit on my own site with Claude Code: crawler access, robots.txt, snippets, markdown and schema. The raw evidence, one real fix, the skill."
url: https://saeedseo.com/blog/ai-seo-audit/
---

# AI SEO Audit With Claude Code: A Real Site Walkthrough

An AI SEO audit asks one question a normal audit doesn't: can ChatGPT, Perplexity and Claude fetch, read and identify you? I ran it on my own site with Claude Code. Here are the requests, the evidence, the one real problem it found, and the skill.

!["Can AI crawlers read my site? AI SEO audit" beside a Claude Code terminal: one saeedseo.com post requested as a browser, GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot and Bingbot, every one answered 200.](https://saeedseo.com/blog/images/ai-seo-audit-768.jpg)

[SEO](https://saeedseo.com/blog/category/seo/) [AI Search](https://saeedseo.com/blog/category/ai-search/)

Published on November 5, 2026 7 min read

[Saeed Shahriar](https://saeedseo.com/about/)

On 4 November I gave Claude Code one job: **check whether ChatGPT, Perplexity, Claude and Google can actually read my site.** Every check had to leave its raw evidence in a file.

Fourteen user agents and sixteen URLs later, **the site mostly passed.** But all six of my published posts had the same small fault, and I'd never noticed it.

The short version: **an AI SEO audit is your technical audit plus a few checks** on access, snippets, reading cost and identity.

Each check proves less than it looks. So the evidence and its limits sit next to every finding.

Just want the steps? Jump to the checklist, or grab the skill file.

## Key Takeaways

-   **Access, reading and being cited are three questions.** This audit covers two.
-   A 200 for a fake "GPTBot" request proves how your server answers a header. **It doesn't prove the real crawler gets in,** and it says nothing about whether you'll be cited.
-   robots.txt is only half of access. **My CDN blocked a common user agent on every path I tested,** robots.txt included.
-   Google needs **no special files or schema.**
-   The real find: **a newsletter heading opened the outline of every post,** before the article's first section.

## What an AI SEO audit checks

**A normal SEO audit asks whether Google can crawl and index you.** The AI layer adds other crawlers, with their own names and their own rules, that you can block without touching Google.

| Question | What answers it | In this audit? |
| --- | --- | --- |
| Can the assistants fetch the page? | Requests per user agent, robots.txt, the CDN's bot rules | Yes, except the CDN dashboard |
| Can they read it, and cheaply? | Snippet controls, the heading outline, a markdown copy | Yes |
| Do they know who wrote it? | Structured data, author links | Yes |
| Do they cite it? | Real questions, [asked many times](https://saeedseo.com/blog/ai-answer-stability/#how-many-runs) | No: [that's a separate test](https://saeedseo.com/blog/geo-vs-seo/#how_i_tested_it) |

Google's part is the simplest. Its guide to [AI features](https://developers.google.com/search/docs/appearance/ai-features) says a page has to be indexed and eligible for a snippet. **Nothing extra.**

## My setup, and what it can't show

|  |  |
| --- | --- |
| Site | saeedseo.com, a static Next.js site on a Cloudflare Worker |
| When | 4 November 2026, around midnight Bali time |
| Pages | All 16 URLs in the sitemap; the home page and one post for the crawler test |
| Tool | Claude Code sending plain HTTP requests, so you can repeat each one with `curl` |

**What it can't show:** whether the real crawlers visit.

OpenAI publishes its crawlers' IP ranges, so real visits can be checked in server logs or the CDN dashboard. **Anthropic doesn't publish its ranges.** This run didn't check logs.

## The audit, in order

### 1\. Request the same page as each crawler

Ask for the page with each crawler's user-agent string. **A different status, or a different body, means something treats that bot differently.**

[![Terminal output: one saeedseo.com blog post requested as a browser, GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot and Bingbot. Every row returns 200 and the same 78,265 bytes.](https://saeedseo.com/blog/images/ai-seo-audit-crawler-test-500.jpg)](https://saeedseo.com/blog/images/ai-seo-audit-crawler-test.jpg)

I sent each crawler's user agent, on the home page and one post.

**Google-Extended isn't in the list on purpose.** It's a robots.txt token for AI training, and it never visits.

### 2\. Read robots.txt the way a crawler does

\*\*A named group replaces the `*` group for that bot.\*\*

Google's [robots.txt spec](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) says it plainly, and it catches a lot of sites, because most people expect the general rules to apply to every crawler and only add the exceptions.

Write `User-agent: GPTBot` with only `Allow: /` under it, and **GPTBot ignores every `Disallow` you wrote for everyone else.**

So the audit lists each group, and flags one that's missing lines the `*` group has.

### 3\. Check snippet and index controls

**No snippet, no AI Overview.** The audit reads each page's robots meta tag, the `X-Robots-Tag` header, the canonical, and any `nosnippet`, `max-snippet` or `data-nosnippet`.

### 4\. Ask for a markdown copy

Request the page with `Accept: text/markdown`. **Some sites, mine included, answer with a plain markdown version.**

[![Terminal output: requesting the GEO vs SEO post with Accept text/markdown returns HTTP 200 with content-type text/markdown, a canonical Link header and Vary: Accept. The markdown copy is 18,322 bytes; the HTML is 78,265.](https://saeedseo.com/blog/images/ai-seo-audit-markdown-copy-500.jpg)](https://saeedseo.com/blog/images/ai-seo-audit-markdown-copy.jpg)

**Nobody has shown that an assistant prefers it.** I list it as an observation, never as a requirement, and I'd be wary of any audit tool that scores you down for not having one.

### 5\. List the heading outline

Pull every page's H1 and H2s in source order. Does anything that isn't the article come first?

### 6\. Read the structured data

List the schema types per page. For the Person behind the site, **check that every post's author points at the same `@id`.** [Here's how I set mine up](https://saeedseo.com/blog/entity-seo-for-llms/), with the commits.

## What I found on saeedseo.com

Here's the crawler test, on the home page and on a post:

| User agent | Home page | Blog post |
| --- | --- | --- |
| Browser | 200 | 200 |
| GPTBot, OAI-SearchBot, ChatGPT-User | 200 | 200 |
| PerplexityBot, Perplexity-User | 200 | 200 |
| ClaudeBot, Claude-User, Claude-SearchBot | 200 | 200 |
| Googlebot, Bingbot, Applebot | 200 | 200 |
| CCBot, Bytespider | 200 | 200 |

Boring? That's what you want here. **Every agent got the same bytes as the browser** (124 KB for the home page, 78 KB for the post), so nothing was cloaked or challenged.

Then a surprise from my own tooling. When Claude Code fetched the sitemap with Python's built-in library, **Cloudflare sent back a 403.**

[![Terminal output: three saeedseo.com paths, the home page, robots.txt and llms.txt, each requested twice. Python's default user agent gets 403 on all three; curl gets 200 on all three.](https://saeedseo.com/blog/images/ai-seo-audit-python-urllib-403-500.jpg)](https://saeedseo.com/blog/images/ai-seo-audit-python-urllib-403.jpg)

curl, python-requests and Go all got through. So did a request with no user agent at all.

The named AI crawlers aren't affected. **But a home-made agent left on Python's default is locked out of my robots file.**

The lock is Cloudflare's **Browser Integrity Check,** which I turned on for this site. The 403 says "Error code: 1010", the block it serves when a request's headers look like a script.

The other checks were clean:

| Check | Result on all 16 URLs |
| --- | --- |
| Status | 200 |
| Robots meta | `index, follow`, no `X-Robots-Tag` |
| Canonical | Equal to the URL |
| Snippet blocks | None |
| Markdown copy | Served on every page, 3.5% to 28% of the HTML's size |
| robots.txt | 8 AI user agents named (six crawlers, plus the Google-Extended and Applebot-Extended training tokens), each group repeating the `*` group's three `Disallow` lines |

### The heading that opened every post (fixed)

This was the finding. **All six posts published at the time had "Get my next blog post first" as their first H2,** before "Key Takeaways".

It was the sidebar's newsletter box. **A reader never saw the problem.**

**In the source, a sign-up heading came before the article's first section.** Nothing shows a model was confused by it.

It was still the wrong first heading. At the end, a second newsletter title was an H2 too.

[![Terminal output: the first four H2 headings in the GEO vs SEO post's source. On the live site before the fix, the first is "Get my next blog post first", marked red, before Key Takeaways. On the live site after the fix, the first is Key Takeaways, marked green.](https://saeedseo.com/blog/images/ai-seo-audit-h2-outline-before-after-500.jpg)](https://saeedseo.com/blog/images/ai-seo-audit-h2-outline-before-after.jpg)

The fix was small. **The newsletter and dialog titles became paragraphs,** with the same font and size, still named for screen readers.

"Related Posts" stayed a heading. It really is a section.

**Fixed and live on 5 November 2026.** Every post's first H2 is now "Key Takeaways".

### Observations that need no fix

-   The Person schema has an `@id`, nine profile links, and every post's author points at it.
-   There's an `llms.txt`. Google Search ignores it, so it's here as an observation.

## The script and the Claude Code skill

**The skill is the audit above, written down so it runs the same way every time.** It takes a site, picks up to ten URLs from the sitemap, runs the six checks and writes a short report.

Three rules in it matter more than the checks:

1.  **No evidence, no finding.** Every line points at a file and a row.
2.  **Label everything** as a finding, an observation or a limit.
3.  **Never edit the site.** The audit reports; I decide what changes.

Get the AI SEO audit skill

The complete ai-seo-audit skill, ready to save as `.claude/skills/ai-seo-audit/SKILL.md`. Free, straight to your inbox.

## How I'd verify a fix

1.  Deploy the change.
2.  Re-run the same check on the live URL, with the same command.
3.  Put the date and the new output next to the old one.

**Don't stop at "it works locally."** My outline fix passed on a dev server first, and only counted as fixed once the live HTML said so.

Getting crawled is only the first step to a ChatGPT citation. The rest is [tracing which pages it pulls and cites](https://saeedseo.com/blog/how-to-get-cited-by-chatgpt/#trace).

## AI SEO audit checklist

| Check | Passes when |
| --- | --- |
| Crawler user agents | Each gets the same status and body as a browser |
| CDN bot rules | Verified bots allowed, checked in the dashboard |
| robots.txt groups | No named group drops the `*` group's `Disallow` lines by accident |
| Snippet controls | No `noindex`, `nosnippet` or `max-snippet:0` where you want to be quoted |
| Heading outline | The first H2 in the source belongs to the article |
| Identity | One Person or Organization `@id`, used by every author line |
| Citations | Tested separately, with real questions asked many times |

## Common Questions

### What is an AI SEO audit?

A technical SEO audit with checks for AI assistants added: whether ChatGPT, Perplexity and Claude crawlers can fetch the pages, whether snippets are allowed, how the page outline reads, and whether structured data identifies the author.

### Does a 200 response to a GPTBot user agent mean OpenAI can crawl my site?

No, that alone proves little. A request with a copied user-agent string only shows how the server answers that header. OpenAI says to verify its crawlers by their published IP ranges, so check server logs or the CDN's verified-bot settings too.

### Do I need llms.txt or special schema for Google's AI Overviews?

No. Google's guide for its AI features says a page must be indexed and eligible for a snippet in Google Search, and that no special files or extra structured data are needed.

### Can a robots.txt rule for one AI crawler cancel my other rules?

Yes. Under Google's robots.txt spec, a group naming a specific crawler replaces the general `*` group for that crawler. Any `Disallow` lines you want it to follow must be repeated inside its own group.

[Saeed Shahriar](https://saeedseo.com/about/)

I've worked in marketing for over a decade, 6 years of it leading SEO at Webnus, a WordPress product company in Canada. Now I live in Bali, run Mersa, a marketing agency I co-founded, and spend most of my day in Claude Code.

This blog is my lab notebook. I test things on real sites, write down what happened, and link every number to where it came from.
