---
title: "How Stable Are AI Answers? Same Question, 10 Times"
description: "I asked ChatGPT the same 10 questions 10 times each, in fresh chats, and coded every name it recommended. What came back, what changed, and how many runs you need."
url: https://saeedseo.com/blog/ai-answer-stability/
---

# How Stable Are AI Answers? I Asked ChatGPT the Same 10 Questions 10 Times

One AI answer is an anecdote. I asked ChatGPT 10 questions 10 times each and counted which names kept coming back. 32 of 57 names came back in 8 or more runs, but two answers had the same names only 16% of the time.

![Terminal output: of 57 names ChatGPT recommended for 7 questions, 32 came back in 8 or more of 10 runs, and two answers had the same list 16% of the time](https://saeedseo.com/blog/images/ai-answer-stability-768.jpg)

[AI Search](https://saeedseo.com/blog/category/ai-search/) [Experiments](https://saeedseo.com/blog/category/experiments/)

Published on November 5, 2026 10 min read

[Saeed Shahriar](https://saeedseo.com/about/)

On 24 October I asked ChatGPT one hiring question three times, each in a fresh chat. **Three answers, six different names,** and only one of them in all three.

So I did it properly. Ten questions, ten fresh chats each, every recommended name written down.

The short version: **more than half the names (32 of 57) came back in 8 or more of 10 runs.** 8 showed up only once.

**The full list was another story.** Two answers to the same question had the same names 16% of the time, and the same order 4%.

Want to run the same test on your own brand? Jump to a test you can run today.

## Key Takeaways

-   **32 of the 57 names came back in 8 or more of 10 runs.** 8 appeared once and never again.
-   Two answers to the same list question had **the same names 16% of the time** and the same order 4%. On average they shared about three quarters of their names.
-   **Aesthetic clinics in Bali moved the most:** the lowest overlap, and no two lists matched.
-   "Who should I hire?" got a job title in 9 of 10 answers. Only one named firms.
-   **One browser in Bali, signed out, one afternoon.** Another place or day can give other names.
-   **Track the mention rate across runs.** One answer's position tells you little.
-   **With 10 runs, a 5 of 10 mention rate could sit anywhere from about 24% to 76%.**

## What changed, and what stayed put

**Most names came back a lot or hardly at all.** Here's every name ChatGPT put forward for the seven questions that ask for names, by how many of the 10 runs it appeared in.

A brand that came up under two questions counts once for each:

| Group | Questions | Names seen | All 10 | 8 or 9 | 2 to 7 | Only 1 |
| --- | --- | --- | --- | --- | --- | --- |
| Agencies | 2 | 11 | 3 | 4 | 4 | 0 |
| Tools | 3, 4 | 20 | 9 | 1 | 5 | 5 |
| Local | 5, 6 | 19 | 7 | 2 | 7 | 3 |
| Comparison | 9, 10 | 7 | 4 | 2 | 1 | 0 |
| All | 7 questions | **57** | **23** | **9** | **17** | **8** |

**I use 8 runs as the line for "keeps coming back".** It's my cutoff, not a standard, so the table shows every band.

**Two questions are left out.** The definitions don't ask for names, so I read them for facts (below). The first hiring question mostly named no one, so it gets its own section.

### The list almost never repeated

**The names held, but the full list kept moving.** For the five questions that ask for a list, I compared every pair of answers to the same question, 45 pairs each.

Overlap is the share of names two answers had in common, out of all the names either one gave, averaged over the 45 pairs:

| Question | Same names | Same names, same order | Average overlap | Same first name |
| --- | --- | --- | --- | --- |
| 2\. GEO agencies | 1 of 45 | 0 | 71% | 8 of 10 |
| 3\. AI tracking tools | 1 of 45 | 1 | 68% | 5 of 10 |
| 4\. Keyword research tool | 21 of 45 | 6 | 85% | 8 of 10 |
| 5\. Dentist in Seminyak | 12 of 45 | 2 | 78% | 9 of 10 |
| 6\. Aesthetic clinics in Bali | 0 of 45 | 0 | 65% | 9 of 10 |
| All five | **35 of 225 (16%)** | **9 (4%)** | **73%** |  |

**The aesthetic clinics question shows the pattern best.** Four clinics came up in all 10 answers, and the same one opened 9 of them.

Yet no two of the 10 lists matched. Eight other names rotated in and out around those four.

That's the gap the studies above keep landing on. **Your brand can be in nearly every answer while the answer itself never repeats.**

### The hiring question named almost no one

I asked: "Who should I hire to get my business recommended by ChatGPT?"

**Nine of the 10 answers named a job title instead of a firm:** a GEO or AI search visibility specialist, with a list of skills to look for.

The tenth searched the web and named three agencies. None of the three appeared in any other answer.

The second hiring question, "Which agencies are best at generative engine optimization?", **asked for firms outright.** It got them: 11 names, 7 of them in 8 or more runs.

**Ask for a type, you get a type.** Ask for names, you get names, mostly the same ones.

### The comparisons barely moved

| Question | ChatGPT's pick, in 10 runs |
| --- | --- |
| ChatGPT or Perplexity: which is better for research? | ChatGPT 9, depends 1 |
| Is Semrush or Ahrefs better for a small business? | Depends 6, Semrush 3, Ahrefs 1 |

**ChatGPT chose itself 9 times out of 10.** "Depends" means the answer didn't pick either side.

### Searching didn't settle it

**56 of the 100 answers showed web links or sources,** and 44 showed none. I expected the answers with links to be steadier, since they draw on live pages.

**They weren't, at least not consistently.** On the clinics question the answers without links overlapped more. On the agencies question, the ones with links did.

With 3 to 7 answers in each half, I wouldn't read anything into either.

A search with links still cites only a few pages. In [a later trace](https://saeedseo.com/blog/how-to-get-cited-by-chatgpt/#what-my-runs-showed), ChatGPT pulled 14 pages for one answer and linked 3.

### The definitions said the same thing

Both definition answers were steady from run to run. **All 10 llms.txt answers carried the same caveat.**

Seven called it a proposed convention. Three said there's no guarantee AI tools read it.

**None of the 10 GEO answers named a source** or said where the term came from. Four of them searched the web and still didn't.

## How I ran it

|  |  |
| --- | --- |
| Assistant | ChatGPT on the web, signed out, its default model for signed-out visitors that day (I didn't record which) |
| Where | Brave on a Mac in Bali, a fresh browser profile, no account, no memory |
| When | 5 November 2026, 10:21 to 16:53 Bali time |
| Questions | 10, in five pairs (below) |
| Runs | 10 per question, each in a new chat, 100 answers in all |
| Pacing | One question at a time, 2 to 4 minutes apart, the order shuffled each round |

**I spaced the questions out on purpose.** A signed-out session that fires 100 questions in a row gets throttled, and a throttled answer isn't the answer a person gets.

Here are the ten questions, exactly as I typed them:

| # | Question | Group |
| --- | --- | --- |
| 1 | Who should I hire to get my business recommended by ChatGPT? | Hiring |
| 2 | Which agencies are best at generative engine optimization? | Hiring |
| 3 | What are the best tools to track if my brand shows up in AI answers? | Tools |
| 4 | What's the best keyword research tool for a small business? | Tools |
| 5 | Can you recommend a good dentist in Seminyak, Bali? | Local |
| 6 | Which aesthetic clinics in Bali do people trust most? | Local |
| 7 | What is generative engine optimization? | Definition |
| 8 | What is an llms.txt file? | Definition |
| 9 | Is Semrush or Ahrefs better for a small business? | Comparison |
| 10 | ChatGPT or Perplexity: which is better for research? | Comparison |

**Then every answer got coded the same way.** For each one I listed the names it put forward as options, in the order they appeared:

1.  Only names of the kind the question asks for. A clinic question counts clinics, and the fillers or lasers they offer are left out.
2.  One spelling per name.
3.  For the two "A or B" questions, the side the answer picked, or "depends" when it refused to pick.
4.  Whether the answer showed web links or sources. That's my stand-in for "searched": a search with no visible links would count as no search.

Claude Sonnet did the coding from those rules, one question at a time.

**Then I checked all 390 names it coded against their answers,** and each one was there.

**That only proves the names are real, not that none were missed.** So Codex hand-checked six answers for missed names and found none.

I also read the 10 hiring answers in full, because 9 of them coded as empty.

**No second coder went through all 100 answers,** so there's no agreement score. Treat the counts as one careful coder's, checked.

I'm leaving the clinics, dentists and agencies unnamed. **They're counted here, and that's all.** A name in an answer is ChatGPT's pick, and it tells you nothing about a clinic's safety.

## "Stable" depends on what you measure

**The studies on this topic each pick a different number,** and then they get quoted side by side as if they agree or disagree. They mostly measure different things:

| Study | What they did | What they measured | Result |
| --- | --- | --- | --- |
| [SparkToro and Gumshoe](https://www.searchenginejournal.com/ai-recommendations-change-with-nearly-every-query-sparktoro/566242/), January 2026 | 12 prompts, 60 to 100 runs each, ChatGPT, Claude and Google's AI answers | Same brand list, and same order | Same list under 1% of the time, same order under 0.1% |
| [Detailed](https://detailed.com/drift/), February report | 5 questions, 100+ runs each, ChatGPT web from the US | Brands that stay across answers; same brands; same brands in the same order | 86%; 23%; 7% |
| [8ight](https://8ight.ai/research/how-ai-recommends-local-businesses/), October 2026 | 267 local questions asked twice, a median 39 minutes apart, ChatGPT API | Of the businesses named in either answer, the share named in both | A median 29%, and none in common for 9% of pairs |
| This test | 10 questions, 10 runs, ChatGPT web signed out | Mention rate per name, list overlap, exact repeats | 32 of 57 names in 8+ runs; same names 16%, same order 4% |

**SparkToro's study was co-written with a company that sells AI tracking,** which they disclosed. 8ight asked through the API, a different route from the app people type into.

Those numbers fit together better than they look. **A brand can come back almost every time while the names around it keep changing.**

That's how "86%" and "under 1%" can both be true.

And stable isn't the same as right. **ChatGPT named itself the better research tool in 9 of 10 runs.**

Maybe it is. **A steady answer tells you what the model says.** Whether that's true is a separate question.

## Why your next answer might differ

**OpenAI doesn't publish a reason any given answer changes,** so I won't guess at the settings inside ChatGPT. What it does document is enough to explain most of what I saw:

-   **It writes a new answer every time.** The model generates each reply, so the same question can come back worded and ordered differently.
-   It decides when to search. 56 of my 100 answers came with web links or sources. The other 44 showed none.
-   Location shapes search results. OpenAI's [ChatGPT search help page](https://help.openai.com/en/articles/9237897-chatgpt-search) says it uses your approximate location from your IP. Mine all came from Bali.
-   **Signed in, memory can steer the search too,** the same page says. I was signed out.
-   **The product changes under you.**

**Which pages it cites is [another test](https://saeedseo.com/blog/how-perplexity-picks-sources/).**

**There's one exception worth knowing.** Asked for a random number between 1 and 100, [Claude said 47 or 42 almost every time](https://saeedseo.com/blog/reddit-post-one-million-views/) across 221 runs.

**With no search and no list, a model's habit can be very steady.** The instability shows up once it's choosing between many real options.

## How many runs are enough?

**Fewer than you'd think will tell you anything, and more than most people run.**

**Here's the range a true mention rate could sit in** when a name shows up in half your runs. It's a 95% Wilson interval, and it assumes each run is independent:

| Runs | Named in half of them | The rate could be |
| --- | --- | --- |
| 5 | 2 or 3 | about 17% to 83% |
| 10 | 5 | about 24% to 76% |
| 25 | about 12 | about 32% to 68% |
| 50 | 25 | about 37% to 63% |
| 100 | 50 | about 40% to 60% |

So a brand named in 5 of my 10 runs could really sit anywhere from a quarter to three quarters of answers, which is a wide range to make a decision on.

**Ten runs show which names keep coming back and which turned up once.** They don't separate 40% from 60%.

**Two things make it worse in practice.** Runs from one browser, one place and one afternoon aren't fully independent.

**Mine also ran over six hours,** so a change between round 1 and round 10 could be the time of day as much as chance. And a rate measured this week says little about next month.

## A test you can run today

1.  **Write 5 questions a customer would type.**
2.  **Ask each one 10 times,** a new chat every time, signed out or in a private window, a few minutes apart.
3.  **Write down every name in each answer,** in order, and whether the answer showed web links.
4.  **Count runs.** For each name: in how many of the 10 answers did it appear?
5.  **Read the rate with the table above.** 9 of 10 keeps coming back. 1 of 10 may be chance.
6.  **Repeat it next month** from the same place, with the same questions.

That's 50 answers, about two hours with gaps. If your name shows up in 2 of 10, **don't celebrate one screenshot** from the run where it did.

If it hardly shows up at all, look at what [GEO changes compared with SEO](https://saeedseo.com/blog/geo-vs-seo/), then run [an AI SEO audit](https://saeedseo.com/blog/ai-seo-audit/) on the pages you want cited.

## Common Questions

### Why does ChatGPT give a different answer to the same question?

It generates each answer fresh, and it decides per answer whether to search the web. Location from your IP, saved memories when signed in, and product updates also change what it says.

### How many times should I ask an AI the same question to measure my brand's visibility?

At least 10 times per question, in new chats. Ten runs show which names keep coming back and which appeared once. To tell a 40% mention rate from 60%, you need closer to 100 runs.

### Does ChatGPT recommend the same brands every time?

Mostly the same brands, rarely the same list. In my test, 32 of 57 names came back in 8 or more of 10 runs, but two answers to one question had identical names only 16% of the time.

### What should I track instead of my brand's position in an AI answer?

The mention rate: out of 10 or more fresh answers to the same question, how many named you. Studies by SparkToro and Detailed found the full list and its order almost never repeat, while how often a brand appears holds steadier.

[Saeed Shahriar](https://saeedseo.com/about/)

I've worked in marketing for over a decade, 6 years of it leading SEO at Webnus, a WordPress product company in Canada. Now I live in Bali, run Mersa, a marketing agency I co-founded, and spend most of my day in Claude Code.

This blog is my lab notebook. I test things on real sites, write down what happened, and link every number to where it came from.
