Skip to content

AI SEO Course: From Search Engines to AI Search

Learn what Google says about how AI Overviews and AI Mode choose their sources, what has to be true technically for a page to be eligible, and how to measure whether any of it worked. Every claim is sourced, and each lesson ends with what was changed on this website and what happened next, including when nothing did.

Price
Free
Signup
None
Certificate
None
Format
Written lessons
Lessons
35 published
Modules
4 of 9 published

Who this course is for

It is written for someone who already knows what a title tag is and wants to know what AI search changes. Nothing here assumes more than that, and nothing pretends the answers are settled.

Written for

  • Marketers and SEOs who know the basics and want to know what AI search actually changes.
  • Developers building sites in JavaScript frameworks.
  • Founders doing their own SEO.
  • Anyone who would rather read the source than a confident summary of it.

Not written for

  • Complete beginners with no SEO exposure. Module 0 is the one for that, and it is not published yet.
  • Anyone who needs a certificate. There is none.
  • Anyone who wants guaranteed rankings. Nobody can honestly promise those, this site included.

Curriculum

How search works

  1. 00

    SEO foundationsPlanned

    Crawling, indexing and ranking, search intent and relevance, how sites earn links and mentions, and how Google Search differs from AI search.

  2. 01

    How search is changingIn progress, 8 of 8 lessons

    AI Overviews and AI Mode next to the classic results page, query fan-out in Google’s own words, and what is and is not known about how sources get cited.


Doing the work

  1. 02

    AI-assisted SEOPlanned

    Where AI genuinely helps an SEO workflow, from research and briefs to technical checks and code, and where it turns into scaled low-value content.

  2. 03

    Keyword and intent researchPlanned

    Keyword research, intent, related questions and entities, worked through on the real results page for “AI SEO course”.

  3. 04

    Content engineeringPlanned

    A content process that adds something instead of recombining existing pages: research, evidence, drafting, citations and review.

  4. 05

    AEO and AI searchIn progress, 8 of 8 lessons

    What AEO, GEO and LLMO mean, what Google calls the same work, and the difference between being cited, mentioned and recommended.


Proving it on this site

  1. 06

    Technical AI SEOIn progress, 12 of 12 lessons

    Crawling, indexing, canonicals, robots.txt, sitemaps, rendering, structured data, performance and AI crawler access, with llms.txt given Google’s stated position.

  2. 07

    Measuring AI search visibilityIn progress, 7 of 7 lessons

    Search Console and its Generative AI performance report, AI referral traffic, sampling AI answers by hand, and the limits of each.

  3. 08

    The 30-day experimentPlanned

    Where this domain started, what was changed, what happened, and what it does and does not prove. Written after the experiment closes.

Thirty-five lessons are published so far

The plan for this course was published before the lessons existed, so you could see what was coming and hold the finished thing to it. Module 1 now has all eight of its lessons written, from AI Overviews through discovery versus ranking. Module 5 has all eight AEO and AI search lessons, from what the acronyms mean through measuring AI visibility. Module 6 has all twelve technical lessons, from crawlability through to llms.txt. Module 7 has all seven measurement lessons, from setting up Search Console through designing an experiment that can fail. Every other module is marked Planned until its lessons are published.

Lessons are written as the experiment runs, so each one can carry a real change made to this website and whatever actually happened next. That is slower than writing it all at once. It is also the only way the examples can be true.

Curriculum updated on . The first round of research changed the plan. There are still nine modules, but two are new: AI-assisted SEO, and AEO and AI search. The old module on how search works is now two, SEO foundations and how search is changing. Three earlier modules no longer stand alone. How the course works is explained on this page, structured data moved into Technical AI SEO, and authority became a lesson in SEO foundations.

Correction, . Earlier versions of this page said Search Console was not yet part of this experiment’s measurement record. The property was verified on 20 September 2026, and the sitemap was subsequently submitted and successfully read. Search Console is now part of the experiment’s record. This does not mean historical Search Console data exists for the earlier experiment days.

How this course compares

Evidence: Observation Four of the courses that rank for “AI SEO course” were fetched and read on . The Semrush, Coursera and HubSpot pages were read again on , and the table shows what they said then. Below is what each one published about itself, beside what this course publishes. It is a record of what those pages said, not a verdict on which course is better for you.

CourseCost, as statedWhat you getSize, as publishedFirst-party experiment
This courseFree, no signupWritten lessons, no video35 lessons, four of nine modulesYes. This website is the subject
Semrush Academy, AI Visibility EssentialsFree, certificate included21 video lessons, built around Semrush tools87 minutes, the sum of its 21 stated lesson lengthsNone stated
Coursera (SkillUp), AI SEO: Mastering GEOFree to enrol, certificate paidTwo modules, videos and four assignmentsAbout four hoursNone stated
Udemy, AI SEO Course 2026Paid, amount not verifiedVideo lectures, mostly classic SEO (secondhand)245 lectures, about 15 hours (secondhand)None stated
HubSpot, AI SEO MasterclassFree behind a formA resource download, not a courseAbout 310 words on the pageNone stated

What this table is not. Every row describes a landing page, not a full curriculum. “None stated” means that page did not say, which is not the same as the course not having one. Prices and contents change, so check each page before deciding anything.

Figures marked secondhand, now only in the Udemy row, come from Charles Floate’s comparison page, modified . He sells a competing course and discloses it. Udemy blocks automated requests, so that row could not be read directly. Two of his claims failed our own checks, which is why the rest is marked rather than repeated as fact.

The one row that is first-party is this course’s own. What has been changed on this site, when, and what happened next is published in full under experiments, including the changes that produced nothing.

Shape of a lesson

Every lesson runs the same six steps, so you always know where the teaching stops and the evidence starts.

  1. ConceptWhat the thing is, in plain language.
  2. Why it mattersWhat changes if you get it wrong.
  3. How to do itSteps you can follow today.
  4. What we changedThe same change, made on this website.
  5. What happenedThe measurement, dated, with its limits.
  6. What we learnedIncluding when the answer is nothing yet.

Module 1: How Search Is Changing

What changed when AI entered the search experience. Lessons are published one at a time; 8 of 8 so far.

Lesson 1.1

AI Overviews: a Search feature, not a separate search engine

Published . Sources checked .

AI Overviews are Google Search responses that can appear above or alongside classic results for some queries. They give a short answer and links for further reading. Google rolled them out to everyone in the United States on 14 May 2024. Evidence: Documented by Google

When does an AI Overview appear?

Google says it designed AI Overviews for queries where they add something beyond classic Search, and that they often do not trigger. That means an Overview is not a fixed result type that every query receives. It is also why one searcher’s screenshot cannot tell us how often Google shows one. Evidence: Documented by Google

What do the links mean?

Google calls these links supporting web resources. A displayed link shows that Google associated that page with this response at that moment. It does not tell us the page’s ordinary rank, the internal reason it was selected, or whether it will appear next time. Those are different questions. Evidence: Documented by Google

Google announced expansion to more than 200 countries and territories and over 40 languages in May 2025. Its 2026 updates report more than 2.5 billion monthly active users. Treat both as dated product announcements, not as a claim about how frequently an Overview appears for any particular keyword. Evidence: Documented by Google Evidence: Documented by Google

Use this lesson

When you see an AI Overview, separate two observations: Google showed the Overview, and Google displayed a supporting link. Do not turn either one into a claim about a page’s normal rank or a guaranteed traffic outcome.

Lesson 1.2

AI Mode: a conversational way to continue searching

Published . Sources checked .

AI Mode is Google’s more conversational Search experience for questions that need exploration, reasoning, or complex comparisons. It gives an AI-powered response with links to supporting websites and invites follow-up questions. Evidence: Documented by Google

How is it related to AI Overviews?

They are related Search experiences, but Google says they may use different models and techniques. So their answers and links can vary. Google also says a reader can ask a follow-up from an AI Overview and continue into AI Mode; that does not make the two experiences mechanically identical. Evidence: Documented by Google

A useful timeline

Google announced AI Mode’s expansion to more than 200 countries and territories on 7 October 2025. In May 2026, it reported that AI Mode had surpassed one billion monthly users. These milestones describe product expansion, not an availability guarantee for every account, country, language, or interface. Evidence: Documented by Google Evidence: Documented by Google

For measurement, a follow-up question in AI Mode counts as a new query in Search Console. That is a reporting rule, not evidence that every follow-up uses a particular retrieval process. Evidence: Documented by Google

Use this lesson

Treat an AI Mode follow-up as a new Search interaction when you read Search Console data. When comparing it with an AI Overview, record each interface and its visible links separately instead of assuming one source list predicts the other.

Lesson 1.3

Traditional SERPs still do the underlying work

Published . Sources checked .

Google describes Search in three stages: crawling, indexing, and serving results. Not every page reaches every stage. AI features did not remove that sequence; they were added to a Search system that still discovers pages, understands them, and returns relevant results programmatically. Evidence: Documented by Google

Why classic results still matter

Google says its generative Search features are rooted in its core ranking and quality systems. It also says normal SEO practices remain relevant for AI Overviews and AI Mode. That is not a promise of visibility. It means the basics of a crawlable, indexable, useful page still matter before an AI feature can show a supporting link. Evidence: Documented by Google

Read Search Console position carefully

An AI Overview occupies one reported position in Search Console, and every link inside that Overview receives the same position. That number describes the Overview container in Google’s report. It is not an ordinary organic rank for each linked page. Evidence: Documented by Google

Use this lesson

Check the state you are discussing before drawing a conclusion: discovered, indexed, served, ordinary rank, or AI-feature display. Search Console position for an AI Overview describes the container, so do not assign that position to every linked page.

Lesson 1.4

Generative answers: retrieval and grounding

Published . Sources checked .

This lesson compares product documentation, not hidden architectures. Evidence: Hypothesis A generative answer may combine earlier model knowledge with retrieved current information.

Google’s documented approach

Google calls its generative Search approach retrieval-augmented generation, or grounding. It says core ranking systems retrieve relevant, current pages from its Search index, then its systems review information from those pages to generate an answer with links. This is Google’s description of Google Search; it is not a universal description of every AI product. Evidence: Documented by Google

Other products document related behavior

OpenAI says ChatGPT Search can rewrite a request into targeted queries, retrieve results, and return source links. Evidence: Documented by OpenAI

Anthropic says Claude web search processes multiple sources and gives direct citations. Evidence: Documented by Anthropic

Microsoft says Copilot can derive a Bing query from a prompt and use returned web information to compose a response. Evidence: Documented by Microsoft

Use this lesson

Name the product whenever you describe retrieval or grounding. If you cannot point to that product’s documentation, describe your explanation as a model or leave it out. This avoids borrowing Google’s terminology for another system.

Lesson 1.6

Source selection: eligibility is documented; the formula is not

Published . Sources checked .

For a page to be eligible as a supporting link in Google’s AI Overviews or AI Mode, Google says it must be indexed and eligible for a normal Search snippet. Google’s newer guidance also names a default-on Search Console control that includes a site in generative AI features. Meeting those conditions does not guarantee that Google will crawl, index, serve, or link to the page. Evidence: Documented by Google Evidence: Documented by Google Evidence: Documented by Google

What Google does recommend

Google says there are no extra technical requirements or special schema needed for these features. It recommends ordinary SEO fundamentals and helpful, reliable, people-first content. Those are eligibility and quality directions, not a published citation formula. Evidence: Documented by Google

What remains unknown

Google explicitly cautions that third-party tools cannot access its internal ranking or AI systems. Correlation studies can describe their own datasets, but cannot reveal a general internal source-selection rule. This course keeps the incompatible E4, E5 and E7 overlap figures out of the curriculum. Evidence: Documented by Google

OpenAI similarly says ChatGPT Search uses multiple factors and does not guarantee placement. That tells us to avoid a false cross-platform formula.Evidence: Documented by OpenAI

Use this lesson

Audit the prerequisites you can verify: indexability, snippet eligibility, Search Console inclusion, and whether the page genuinely helps its reader. Then record displayed sources as observations. Do not convert a correlation or a vendor score into a claimed citation factor.

Lesson 1.7

Citations: a displayed source association, not a score

Published . Sources checked .

Evidence: Hypothesis A citation or source link shows an interface-associated web source, but does not by itself explain selection or recurrence.

Google describes AI Overviews and AI Mode as showing links to supporting web resources. In Search Console, a click on an external AI Overview link counts as a click. That reporting rule does not turn the link into a conventional rank. Evidence: Documented by Google

ChatGPT Search responses may include citations and source links, and OpenAI warns that search results and citations can be incomplete, outdated, or incorrect. Claude says its web-search responses include direct citations and source links. Microsoft documents query citations in Copilot Chat that reveal the Bing searches derived from a prompt; those are not the same thing as a page citation. Evidence: Documented by OpenAI Evidence: Documented by Anthropic Evidence: Documented by Microsoft

A useful distinction

Retrieved → cited → mentioned → recommended is this course’s conceptual model, not a documented universal pipeline. Keep the terms separate when measuring visibility.

Use this lesson

For each response you record, note the prompt, date, platform, interface and visible links. Describe only what appeared. Do not infer a ranking, a recommendation, traffic, or the hidden reason a source was selected.

Lesson 1.8

Discovery versus ranking: different states, different evidence

Published . Sources checked .

Google discovers URLs through crawling, links and sitemaps. It then decides whether to index them and, later, whether to serve them for a query. A page can therefore be discovered without being indexed, indexed without ranking for a query, or visible in Search without appearing in an AI feature. Evidence: Documented by Google

Eligibility is not display

Google requires indexed, snippet-eligible pages for supporting-link eligibility in AI Overviews and AI Mode. Its Search Console control also lets a site include or exclude its content from those features. Google still says meeting requirements does not guarantee crawling, indexing, serving, or a supporting link. Evidence: Documented by Google Evidence: Documented by Google Evidence: Documented by Google

OpenAI’s rule is separate: it says a site must allow OAI-SearchBot to be eligible for ChatGPT Search inclusion, while placement remains unguaranteed. One platform’s discovery is not evidence of another platform’s discovery.Evidence: Documented by OpenAI

Google’s generative AI performance report records impressions from AI Overviews and AI Mode by page, country and device. Measurement belongs in Module 7; here it reinforces that display is its own state. Evidence: Documented by Google

Use this lesson

When you report visibility, name the state you measured and the platform that produced it. “Indexed in Google,” “shown as an AI Overview supporting link,” and “ranked for an ordinary result” are different observations and need different evidence.

Module 5: AEO and AI Search

What the new acronyms mean, which of them Google actually uses, and how to record what an AI answer shows. Lessons are published one at a time; 8 of 8 so far.

Lesson 5.1

Answer Engine Optimization: an industry label, not a Google discipline

Published . Sources checked .

Answer Engine Optimization, usually shortened to AEO, is an industry label for work meant to make information useful to systems that return a direct answer instead of a list of links. It was first discussed mostly around voice search and answer boxes.

Where the term comes from

A Search Engine Watch article from 7 February 2018 shows the term in use: “This strategy has come to be known as AEO, or ‘answer engine optimization’.” Evidence: Expert / industry Jason Barnard says he coined it: “I coined the term Answer Engine Optimization (AEO) in 2017”. Evidence: Expert / industry That is his own claim. The 2018 article quotes Barnard as one of its sources, so it is not independent confirmation of who coined the term.

What Google calls the same work

Google acknowledges the term without adopting it. Its guide says that “terms like Answer Engine Optimization (AEO) or Generative Engine Optimization (GEO) are common online”, and that optimizing for its generative AI features “is optimizing for the search experience, and thus still SEO.” The features it means are AI Overviews and AI Mode. Evidence: Documented by Google

Use this lesson

When someone recommends an AEO tactic, write down four things: which platform it is about, which visible outcome it claims to change, what evidence is offered, and what result would show it does not work. Then turn the advice into a question about the reader. “Add a direct answer” becomes “can a reader understand this answer without searching elsewhere, and is it accurate?” That is a useful check on its own. It is not evidence of more visibility.

Evidence: Hypothesis Any claim that a specific AEO tactic raises visibility stays a hypothesis until a source documents it or a controlled test supports it.

Lesson 5.2

GEO and LLMO: keep the source and the setup attached

Published . Sources checked .

Generative Engine Optimization (GEO) and Large Language Model Optimization (LLMO) are labels for work aimed at visibility in generative search and language-model answers. They overlap with AEO and with each other, and none of them names one standardized process. Google treats this work as SEO, as lesson 5.1 showed. Evidence: Documented by Google

GEO: a research paper

The term GEO is tied to a paper, GEO: Generative Engine Optimization, by Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan and Ameet Deshpande. It was first submitted to arXiv on 16 November 2023, and version 3 was revised on 28 June 2024. Evidence: Expert / industry The paper introduces the GEO framework and a benchmark, GEO-bench. That gives the term a source. It does not make GEO an official ranking system or a settled industry standard.

The paper reports results from two setups, and they should not be mixed. The main experiments use the authors’ own generative engine, built on GPT-3.5-turbo and evaluated on GEO-bench. A separate test ran on Perplexity.ai, where the authors report that their best-performing methods “improve upon baseline by 22% on Position-Adjusted Word Count and 37% on Subjective Impression”. Those are the paper’s own measures. They are not a traffic lift or a citation rate for a website. The content was rewritten by a language model, and the systems tested date from 2023 and 2024. Evidence: Expert / industry

LLMO: an attributed coinage

In a post dated 15 December 2022, Han Xiao and Alex C-G of Jina AI wrote: “LLMO, or large language model optimization, is a term we coined”. Evidence: Expert / industry A WordLift article from 21 December 2022 repeats the claim and credits them. Evidence: Expert / industry That is a claim of coinage made by the authors themselves. Nobody has shown it was the first use.

Use this lesson

For every GEO or LLMO claim, record who makes it, which system was studied, the date, what it was compared against, the metric, the result, and the limits. Keep each setup in its own row. Never merge two results because they share an acronym or a paper. A circulating table that puts “+40%”, “+37%” and “+30%” side by side as Perplexity results does exactly that, and this course does not use it.

Lesson 5.3

Citations, mentions and recommendations are different observations

Published . Sources checked .

Lesson 1.7 established that a citation is a displayed source association, not a score. Read it first. This lesson adds the part that matters for AEO work: sorting one answer into separate observations, because most AI SEO advice treats them as one.

Five terms, kept apart

Mention
The answer names a person, brand, site or product, with or without a link.
Citation
The answer visibly attaches a source to the response, usually as a link, card or marker.
Cited URL
The exact address the citation opens. A domain named in the prose is not a cited URL.
Recommendation
The answer advises choosing, using or considering something. A citation alone is not a recommendation.
Retrieved
The platform fetched or used a page behind the scenes. You usually cannot see this in the answer.

These are this course’s categories, not a documented pipeline that every platform follows. What the platforms do document is narrower. Google’s AI features give a response “with links to supporting websites”. Evidence: Documented by Google OpenAI says responses that use web search “may include citations”, and that “search results and citations can be incomplete, outdated, or incorrect.” Evidence: Documented by OpenAI Anthropic says Claude’s web-search responses include citations and source links. Evidence: Documented by Anthropic

A worked example

This example is invented to show the classification. It is not an observation. “You can learn this at Example Academy” is a mention. A linked source card pointing to example.com/lesson is a citation, and that address is the cited URL. “I recommend Example Academy” is a recommendation. One answer can contain all three, and each is recorded on its own.

Use this lesson

For each answer, save the prompt and the full response. Mark every entity mentioned, every citation, the exact URL behind each one, and any explicit advice to choose something. Write “not visible” when a state cannot be told from the answer. A mention rate is not a citation rate, and a citation does not show why the source was chosen or whether the answer endorses it.

Lesson 5.4

Entity relevance: make references clear to people

Published . Sources checked .

An entity is a distinct person, organization, product, place or idea that a page refers to. This lesson uses entity relevance as an editing check: can a reader tell which entity a name refers to, how it relates to the topic, and what supports that relationship?

What Google documents, and what it does not

Google’s AI features page says: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.” Evidence: Documented by Google Its newer optimization guide adds one condition, the Search Console control covered in lesson 5.5. It also says: “You don’t need to write in a specific way just for generative AI search.” Neither page mentions entities or an entity score. Evidence: Documented by Google

Evidence: Hypothesis Clearer entity references change how often AI answers cite or recommend a page. Untested here. It would be weakened if a fixed prompt sample showed no change after a controlled edit, though one small sample could not settle it either way.

How to check a page

  1. List the names and organizations that carry meaning on the page.
  2. Identify each at first mention, in plain words. Separate people or products with similar names.
  3. State a relationship only when a source supports it, and link that source.
  4. Check that names and roles match across the page and the site’s about page.
  5. Do not add profiles, affiliations or titles you cannot show to be true.

What we changed

On 29 September 2026 this site’s about page added two external profiles for the author, GitHub and LinkedIn. Both are shown as visible links, and the page’s structured data asserts them with sameAs, built from the same list so the two cannot drift apart. Nothing else was added: no job title the site cannot evidence, no organization, no credential. The GitHub profile was checked and links to the LinkedIn profile. LinkedIn blocks automated checks, so that link rests on the owner’s confirmation.

What happened

Nothing was measured for this change on its own. It shipped during another experiment on this site and was logged there as a confounder. That experiment was ended early, on 5 October 2026, without a result. So there is no evidence here, either way, that the entity work changed anything in search or AI answers.

Lesson 5.5

Source selection: what each platform documents

Published . Sources checked .

Lesson 1.6 covers Google’s eligibility conditions in full. Start there. This lesson puts the documented conditions for each platform side by side, so an AEO claim can be checked against what that platform actually says.

Google: two pages, two dates

To be a supporting link in AI Overviews or AI Mode, “a page must be indexed and eligible to be shown in Google Search with a snippet”. The same page, last updated December 2025, says “There are no additional technical requirements.” Evidence: Documented by Google

Google’s optimization guide, updated July 2026, adds a condition: “a site must be included in Search generative AI features in Search Console to be eligible”. Evidence: Documented by Google That control is on by default: including a site’s links and content “is the default control for all properties.” Evidence: Documented by Google So most sites already meet the condition. Only a site that has opted out has not.

None of these conditions is a guarantee. Google says plainly that “indexing and serving isn’t guaranteed.” Evidence: Documented by Google

OpenAI and Anthropic

OpenAI says web-search responses “may include citations”, and tells users to “Select Sources, when available, to view cited sources and other relevant links.” Evidence: Documented by OpenAI Anthropic documents citations and source links in Claude’s web-search answers. Evidence: Documented by Anthropic Both describe what a user sees. Neither publishes how a given source is chosen.

Use this lesson

When a page is missing from an answer, check only what you can observe: the page is public, indexable, snippet-eligible, and included in Google’s generative AI control. Then repeat the same prompt under recorded conditions. A single answer is not a selection rule.

Write every causal explanation as a hypothesis with a way to fail. For example: “If the page is easier to understand, it is cited more often across this fixed prompt set.” No change, or a drop, across a predeclared sample would fail to support it. Even a rise would not reveal the platform’s internal reasons.

Lesson 5.6

Factual clarity: make each important claim checkable

Published . Sources checked .

Factual clarity means a reader can tell what is claimed, about what, under which conditions, and where the evidence comes from. It does not mean short sentences, and no format guarantees a place in an AI answer.

Google’s helpful content guidance asks questions such as “Does the content provide substantial value when compared to other pages in search results?” and whether it “clearly demonstrate[s] first-hand expertise and a depth of knowledge”. Those are broad principles. They are not a formula linking clarity to AI citations. Evidence: Documented by Google

How this course does it

Every claim on this page carries one of five evidence labels: documented by a named publisher, expert or industry, first-party experiment, observation, or hypothesis. A documented claim always names who documented it, so OpenAI’s documentation is never shown as Google’s. A conclusion drawn from evidence stays written as a conclusion. It is never promoted to a documented fact.

Use this lesson

Audit a paragraph with four questions:

  1. What is the factual claim, in one sentence?
  2. Which source directly supports it, and is that source about the same platform and date?
  3. Which part is interpretation or hypothesis?
  4. What qualification stops a reader from applying it too broadly?

For example, “Clear facts make AI systems cite you” becomes “Clear sourcing helps a reader check this explanation. Whether it changes how often AI answers cite the page is a hypothesis, to be tested by comparing a fixed sample before and after one controlled change.” The first sentence is a reason that holds for readers. The second names an untested idea. Neither pretends to be a Google rule.

Lesson 5.7

Passage-level usefulness without chunking myths

Published . Sources checked .

A useful passage answers a real sub-question accurately and carries enough context that someone reading only that passage understands its subject, conditions and source. That is a writing standard. It does not establish that AI systems retrieve text in fixed chunks, or that any passage length is best.

Google is direct about this for its own features: “There’s no requirement to break your content into tiny pieces for AI to better understand it. Google systems are able to understand the nuance of multiple topics on a page and show the relevant piece to users.” It adds that “there’s no ideal page length”. Evidence: Documented by Google That is Google’s guidance for Google Search. It says nothing about how other platforms work.

Why context still matters

A paragraph that opens with “this” or “it” may depend on a sentence several screens earlier. A reader who arrives through a link, an excerpt or a copied note loses that context. Restoring it helps the reader. Whether it changes retrieval or citation is a separate hypothesis.

Use this lesson

  • Does the first sentence name the topic instead of leaning on a vague pronoun?
  • Are scope and conditions stated next to the claim?
  • Can the reader tell evidence from explanation and interpretation?
  • Is the supporting source linked where the claim is made?
  • Can repeated background be cut without making the passage misleading?

Do not force every paragraph into a question-and-answer box, and do not pad text to reach a length. Keep the page’s structure where it helps a reader follow the whole explanation.

Lesson 5.8

Measuring AI visibility: building the sample

Published . Sources checked .

Lesson 7.5 sets out this course’s sampling protocol: three to five runs per prompt and platform on the same day, results written as a rate with its sample size, and the prompt set fixed before the first run. Use it as written. This lesson covers the two inputs that protocol needs: a prompt set and a comparison panel.

What the platform reports can and cannot replace

Google’s Generative AI performance report counts Search impressions for AI Overviews and AI Mode. Its help page says that “as of August 31, 2026, we’ve rolled out these insights to all websites worldwide”, and also that “not all properties have access to the report, as we’re rolling out over time.” A site with too few impressions in these features will not see it. Evidence: Documented by Google

Microsoft’s AI Performance report, launched as a public preview in February 2026, covers citations “across Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations”. Microsoft says its average cited pages metric “does not indicate ranking, authority, or the role of any page within an individual answer.” Evidence: Documented by Microsoft Neither report shows ChatGPT, Claude or Perplexity answers, and neither shows the prompt behind a citation. That gap is what manual sampling fills.

Fix the prompt set first

Choose prompts before you run any, and keep the exact wording across every round. These five are an example set for this course’s topic. They are a designed set, not evidence of what people search for, and they have not been run yet.

An example prompt set. Not yet run.
IDExact prompt
P01What is AI SEO, and what should a beginner learn first?
P02What does generative engine optimization mean?
P03Compare free resources for learning AI SEO.
P04How can a website owner measure whether AI search tools cite a page?
P05What is the difference between an AI search citation and a recommendation?

Treat each surface as its own platform. Google AI Overviews and AI Mode are separate rows. If a surface is not available in your country, device or account, record it as unavailable. Never swap in another service.

Build the comparison panel carefully

A panel is the fixed list of other entities you check for in every answer. For each entry, record what it actually is, not just its name: a course, a listicle, a comparison page, a video, a gated resource. Name the person behind a course on a marketplace, not just the marketplace.

Evidence: Observation On 5 October 2026 we checked an eight-entry draft panel for this course’s topic. Two of the entries were not courses. One was a blog listicle of courses. The other was a scored comparison page, published by someone who sells a competing paid product. Labelled only by name, both would have been counted as courses. The panel has been held back until it is rebuilt with a format for every entry.

Record each answer the same way

Use one row per prompt, platform and run. Record mention, citation, cited URL and recommendation separately, as lesson 5.3 defines them, and add the run conditions lesson 7.5 lists. Write “unclear” or “unavailable” instead of counting a run as a no. Report every rate with its numerator and denominator, for example cited 2/5, n=5. Do not combine platforms or prompts into one score.

Module 6: Technical AI SEO

What has to be true technically before anything else matters. Lessons are published one at a time; 12 of 12 so far.

Lesson 6.1

Crawlability: can search engines and AI tools reach your pages?

Published . Sources checked .

Crawlability is the question of whether a crawler can reach and fetch a page. This lesson covers what “crawlable” means, which parts of your setup can block a crawler, and how to check your own site, including what that check cannot tell you.

You will be able to
Check whether a page can be fetched by search and AI crawlers, including blocks that sit outside robots.txt, and state what your check does and does not prove.
Before this lesson
Module 0, SEO foundations (crawl, index, rank).

Concept

Google describes Search as three stages: crawling, indexing and serving. Not every page gets through each stage. In Google’s words, “Google Search works in three stages, and not all pages make it through each stage”. Evidence: Documented by Google

Crawling is the first stage. A crawler is an automated program that downloads pages. If a crawler never fetches a page, it never reads what is on it.

One way crawlers find new pages is through links. Google: “Other pages are discovered when Google extracts a link from a known page to a new page”. Evidence: Documented by Google

Some pages are not reachable at all. Google again: “Some pages may be disallowed for crawling by the site owner, other pages may not be accessible without logging in to the site.” Evidence: Documented by Google

Why it matters

The same requirement applies to Google’s AI features. Google’s guide on generative AI search says to “ensure your content is crawlable, as Google Search generative AI models use publicly accessible, crawlable content”. Evidence: Documented by Google

Robots.txt is not the only thing that decides whether a crawler gets through. Among its steps for AI features, Google lists “[e]nsuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure”. Evidence: Documented by Google

That matters because a CDN can block a crawler that robots.txt allows. Cloudflare, for example, lets site owners set Block or Allow controls for individual AI crawlers. Evidence: Documented by Cloudflare If your robots.txt says “allow” and your CDN says “block”, the crawler is blocked.

How to do it

  1. Read your robots.txt. Open https://your-site/robots.txt. Lesson 6.4 covers what each line means.
  2. Check your CDN and host. If you use a CDN or a managed host, look for bot controls, firewall rules or “AI crawler” settings. These sit outside robots.txt, and they can block a crawler your robots.txt allows.
  3. Fetch a page as a crawler would identify itself. From a terminal, request a page with a crawler’s user-agent string and look at the status code:
    curl -s -o /dev/null -w '%{http_code}\n' \
      -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
      https://your-site/your-page
    200 means the server returned the page to that request. A different code, such as 403, means the request was refused or handled differently; look at the response to see why.
  4. Write down what you ran and when. If your analytics or CDN counts crawlers by user-agent, your own test requests will be counted as crawler traffic. Note the time window so you can exclude it later.
  5. State the limit of your check. See “What this does not prove” below.

What we changed on this site

Nothing was changed for this lesson. The relevant setup was done on , Day 1 of the experiment:

Correction, . This line first said the setup “dates from Day 2 of the experiment”. That used an earlier work-phase numbering. The site now uses “Day” only for experiment days, counted from 18 September, so the setup belongs to Day 1.

  • robots.txt contains User-agent: * and Allow: /, plus the sitemap location.
  • Pages are server-rendered, so the content is in the HTML a crawler downloads. Lesson 6.6 covers why that matters.
  • The site runs on Vercel behind Cloudflare, so there are two layers besides robots.txt where a crawler could be refused.

What happened

Evidence: Observation

We requested /ai-seo-course once with each of ten crawler user-agent strings: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot and bingbot. The requests ran between 05:25:17 and 05:25:21 UTC.

The site returned HTTP 200 to every one.

What this does not prove

This test shows that the site did not reject those user-agent strings. It does not show that genuine crawlers visited, or that any AI system retrieved, processed, cited or used the page.

The reason is in the test itself. A user-agent string is just text that the requester sends, and we sent all ten from our own computer. The string alone could not tell the site who we were. Cloudflare says the same about its own crawler reports on the free plan: they identify AI crawlers “based on their user agent strings”, and paid plans add “a more thorough detection”. Evidence: Documented by Cloudflare

It also does not show how real crawlers are treated. They arrive from their operators’ own networks, and a CDN may handle those requests differently from ours.

Keep four things apart:

What you can seeWhat it supportsWhat it does not prove
A request carrying an AI platform’s user-agentA request with that string reached the siteThat the platform actually sent it
A request from a user-triggered fetcher (for example ChatGPT-User)A user-triggered fetch may have happenedThat the page was used in an answer
Retrieval for an AI answerNot directly observable from the website’s side
A citation in an AI answerThat answer cited the pageWhy it was chosen, or how often it happens

Modules 5 and 7 return to this table.

Because our own test requests would look like crawler traffic in any user-agent-based report, we logged the time window above so it can be excluded from the experiment’s crawler data.

What we learned

A 200 response to a crawler’s user-agent is a useful first check: it rules out an obvious block on that string. It is the start of an answer, not the answer. Whether real crawlers reach this site, and whether any AI system uses what they fetch, are separate questions. Later lessons measure those separately and label each result with what it can and cannot support.

Terms used

Crawl
A crawler automatically downloading a page.
Fetch
A single request for a page.
User-agent
The name a requester sends to identify itself. The server cannot confirm it from the name alone.
CDN (content delivery network)
A network of servers between visitors and your host, which can also filter requests.
WAF (web application firewall)
Rules that allow or block requests before they reach your site.

Lesson 6.2

Indexing: why a live page still might not be in Google

Published . Sources checked .

A page being live is not the same as a page being in Google, and being in Google is not the same as being eligible to show up in an AI Overview. This lesson covers what indexing is, the three separate things that have to be true before a page can appear as a supporting link in Google’s AI features, and how to tell which of them are true for your own site.

You will be able to
Tell apart crawled, indexed and eligible for AI features, check where your own pages sit, and say which stage you have evidence for and which you are guessing at.
Before this lesson
Lesson 6.1, Crawlability.

Concept

Crawling and indexing are different stages. Google describes indexing like this: “After a page is crawled, Google tries to understand what the page is about. This stage is called indexing and it includes processing and analyzing the textual content and key content tags and attributes”. Evidence: Documented by Google

Getting crawled does not earn you a place in the index. Google is direct about this: “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Evidence: Documented by Google

Among the reasons Google lists for a page not being indexed are that “[t]he quality of the content on page is low”, that “[r]obots meta rules disallow indexing”, and that “[t]he design of the website might make indexing difficult”. Evidence: Documented by Google

During indexing Google also decides which address represents the content. It “determines if a page is a duplicate of another page on the internet or canonical”, and “also collects signals about the canonical page and its contents”. Evidence: Documented by Google Lesson 6.3 covers what you can do about that choice.

Why it matters

Indexed is a precondition for AI features, not a ticket to them

Google states the requirement for its AI features plainly: “To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements.” Evidence: Documented by Google

Read the verb. The page becomes eligible. Nothing in that sentence promises an appearance, and this lesson does not claim one. What it does tell you is the order: a page that is not indexed cannot be a supporting link, so indexing is the gate you have to get through first.

Snippet eligibility is a second, separate condition. Google lists the controls that affect it: “To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls.” Evidence: Documented by Google A page can be indexed and still be made snippet-ineligible by one of those.

Two Google pages disagree, and the dates matter

This is worth slowing down for, because the two pages say different things and both are live.

The AI features page, last updated 10 December 2025, ends its eligibility paragraph with: “There are no additional technical requirements.” Evidence: Documented by Google

The generative AI optimization guide, last updated 10 July 2026, says the opposite: “In addition to the technical requirements for Search, a site must be included in Search generative AI features in Search Console to be eligible for display in generative AI features on Google Search.” Evidence: Documented by Google

The newer page describes a control that did not exist when the older one was written. Google’s Search Console help for that control says: “As of August 31, 2026, we’ve rolled out this control to all websites worldwide.” Evidence: Documented by Google

So the practical reading is: the July 2026 page reflects the current setup, and the December 2025 sentence has not been updated to match. We are not guessing which is right in principle. We are noting which is newer, and that the newer one describes a control with a documented rollout date.

Two things about that control are easy to get wrong.

It is on by default. Google’s help page describes “Include my site’s links and content in Search generative AI features” as the option to which “[t]his is the default control for all properties” applies. Evidence: Documented by Google For a property still using the default, the control is Include.

It is not a ranking control. Google: “This control only affects whether your content can appear in certain Search generative AI features; this control isn’t used as a ranking or inclusion signal affecting other parts of Search.” Evidence: Documented by Google Setting it to include does not help ordinary rankings, and this lesson makes no such claim.

If you do change it, expect a wait: “Once you change your control in Search Console, it generally takes a few days for your site’s content to be excluded from the applicable Search generative AI features.” Evidence: Documented by Google

The noindex trap

If you want a page kept out of the index, noindex is the tool. Google defines it as a rule “set with either a <meta> tag or HTTP response header and is used to prevent indexing content by search engines that support the noindex rule, such as Google.” Evidence: Documented by Google

The trap is combining it with robots.txt. Google: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” Evidence: Documented by Google

And the consequence: “If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results.” Evidence: Documented by Google

Blocking and de-indexing pull in opposite directions. Block the page and the crawler never reads the instruction telling it to drop the page. Lesson 6.4 covers what robots.txt does control.

How to do it

  1. Separate the questions. “Is my page live?”, “has it been crawled?”, “is it indexed?” and “is it eligible for AI features?” are four questions with four different answers. Write them as four rows, not one.
  2. Check live yourself. Request the page and read the status code. That tells you the page is reachable. It tells you nothing about the index.
  3. Use URL Inspection in Search Console. This is the only tool that reports Google’s own view of a URL: whether it is on Google, which canonical Google picked, and when it was last crawled. Nothing you run from your own machine can answer those.
  4. Read the Page indexing report for the site-wide picture, and read the reasons it gives for anything not indexed.
  5. Check the generative AI control under Settings, once. It is on by default, so the check is usually a confirmation rather than a change.
  6. Write down which stage each page has reached, and the date. A page moves through these stages over time. An answer without a date goes stale without telling you.

A warning on the sitemap number. Search Console’s Sitemaps report shows a “Discovered pages” count. That is how many URLs Google read out of your sitemap file. It is not a count of indexed pages, and the two numbers can stay far apart indefinitely.

What we changed on this site

Nothing was changed for this lesson. The setup it describes was done on , Day 3 of the experiment:

  • A Search Console property was verified for https://learnaiseocourse.com/, using the HTML file method. It is a URL-prefix property, not a Domain property, because DNS for this domain is controlled by someone else.
  • The sitemap was submitted.
  • The Search generative AI control was read, not changed.

What happened

Evidence: Observation

Three things are now on the record, and one important thing is not.

StageThis site, on 20 September 2026
Live and reachableYes. All five pages return HTTP 200
Listed in our sitemapYes. 5 URLs
Read by Google from the sitemapYes. Submitted and read 20 September, status Success, 5 pages discovered
CrawledNot recorded
IndexedNot recorded
Included in Search generative AI featuresYes. Control reads Include, inherited default

The sitemap row and the indexed row are the two most often confused, so it is worth being blunt about what the Sitemaps report actually said. It said Google read five URLs out of the file. Five pages discovered is not five pages indexed. No indexing status has been recorded for this site.

What this does not prove

The sitemap being read does not mean any page was crawled. Crawling does not mean indexing. And indexing would not mean an appearance in an AI Overview, for the reason in “Why it matters”: the documented word is eligible.

The generative AI control reading Include tells you one eligibility condition is satisfied. It says nothing about the others, and by Google’s own statement it says nothing at all about ordinary rankings.

There is also a gap in this experiment that no later work can close. Search Console reports from the date a property is verified, and this property was verified on 20 September 2026, three days after the repository’s first commit. There is therefore no Search Console data for the days before that. We have not found a Google page stating the no-backfill behaviour in those words, so we are reporting it as the practical limit we are working under rather than as documented behaviour.

What we learned

The useful mental model is a chain where each link can fail independently:

live -> crawled -> indexed -> snippet-eligible
-> included in the AI control -> eligible to be a supporting link

Eligible is the last word in that chain, and it is where Google’s documentation stops. Everything after it is somebody’s inference.

On this site, we can currently evidence the first link and the last, and not the ones in the middle. That is an uncomfortable place to write a lesson from, and it is also the honest one. When indexing status is recorded, it will be added here as a new dated observation rather than folded into this one.

Terms used

Index
Google’s store of pages it has processed and may serve.
Indexed
Present in that store. Not the same as crawled, and not a promise of being shown.
Snippet eligibility
Whether a page may be shown with a snippet, which Google gives as a precondition for AI Overview supporting links.
noindex
A meta tag or HTTP header asking search engines not to index a page. Only works if the crawler can reach the page to read it.
Search generative AI control
A Search Console setting governing whether a site can appear in Google’s generative AI features. On by default.
Discovered pages
Search Console’s count of URLs read out of a sitemap file. Not an indexing figure.

Lesson 6.3

Canonicalization: which version of a page counts?

Published . Sources checked .

The same content can often be reached at more than one address: with and without www, over http and https, with and without a trailing slash. Canonicalization is how one of those addresses gets treated as the real one. This lesson covers what Google means by a canonical URL, which signals you control, and how this site went from one broken address variant to a working one.

You will be able to
Choose one URL for each piece of content, send consistent signals for it, and check every address variant of your own pages.
Before this lesson
Lesson 6.1, Crawlability. Lesson 6.2, Indexing, is not published yet; this lesson does not depend on it.

Concept

Google defines it like this: “a canonical URL is the URL of a page that Google chose as the most representative from a set of duplicate pages.” Evidence: Documented by Google

Notice who chooses. You can state a preference, but Google says “indicating a canonical preference is a hint, not a rule”, and “Google may choose a different page as canonical than you do, for various reasons.” Evidence: Documented by Google

Duplicate addresses are also not, by themselves, a problem to panic about: “Some duplicate content on a site is normal and it’s not a violation of Google’s spam policies.” Evidence: Documented by Google

Why it matters

If you leave the choice entirely to Google, it may pick an address you did not intend. The signals you send are how you make your preference clear. Google describes them with different strengths:

  • “Redirects: A strong signal that the target of the redirect should become canonical.”
  • “rel="canonical" link annotations: A strong signal that the specified URL should become canonical.”
  • “Sitemap inclusion: A weak signal that helps the URLs that are included in a sitemap become canonical.” Evidence: Documented by Google

Two tools are the wrong ones for this job. Google: “Don’t use the robots.txt file for canonicalization purposes.” And it does not recommend noindex either, “because it will completely block the page from Search”. Evidence: Documented by Google

A note on redirect codes: Google says “The 301 and 308 status codes mean that a page has permanently moved to a new location.” Evidence: Documented by Google This site’s redirects are 308s.

How to do it

  1. Pick one address format. Decide on https, with or without www, and with or without a trailing slash. Write it down.
  2. Check the canonical tag. View the page source and find <link rel="canonical" href="...">. It should point to your chosen address, written in full.
  3. Check every variant. For one page, request each version and look at the status code and where it redirects:
    curl -sI http://example.com/page      | grep -iE '^HTTP|^location'
    curl -sI http://www.example.com/page  | grep -iE '^HTTP|^location'
    curl -sI https://www.example.com/page | grep -iE '^HTTP|^location'
    curl -sI https://example.com/page/    | grep -iE '^HTTP|^location'
    Every variant should either be your chosen address (status 200) or redirect towards it. If a variant does not respond at all, that address is broken for anyone who follows a link to it.
  4. Follow the whole chain. curl -sIL <url> follows every redirect and prints each step. Note how many steps it takes to reach a 200.
  5. Make the signals agree. The canonical tag, the redirect targets and the URLs in your sitemap should all name the same address.

What we changed on this site

Evidence: Observation

Before. The main domain was already consistent: each page’s canonical tag pointed to itself, http redirected to https with a 308, and a trailing slash redirected to the address without it, also with a 308. But www.learnaiseocourse.com had no DNS record at all. It was first found missing in a review on 19 September and confirmed missing again at 05:24:50 UTC. Anyone following a link to the www address got no response.

The change. A DNS record for www was added in Cloudflare. That is the only change on our record. The redirect itself is returned by the host: the www response carries a Vercel header. The record was added between 05:24:50 UTC, when it was last seen missing, and 06:29 UTC, when it was seen working.

What happened

Evidence: Observation

After the record was added, every www address we tested redirected to the same path on the main domain with a 308:

RequestedResponse
https://www.learnaiseocourse.com/308 to https://learnaiseocourse.com/
https://www.learnaiseocourse.com/ai-seo-course308 to https://learnaiseocourse.com/ai-seo-course
http://www.learnaiseocourse.com/ai-seo-course308 to https://www.learnaiseocourse.com/ai-seo-course, then 308 to the main domain
http://www.learnaiseocourse.com/ai-seo-course/Three redirects, then 200 at https://learnaiseocourse.com/ai-seo-course

The longest chain we found was that last one: http to https, www to the main domain, then the trailing slash removed. Three redirects, each a 308, ending at the canonical address.

What this does not show

These results show what the site sends. They do not show which address Google has chosen as canonical. Only Google can report that, through Search Console’s URL Inspection (lesson 6.2 and Module 7). No such report has been recorded for this site yet.

Correction, . This line first said “this site’s Search Console setup is not complete yet”. That stated the account’s state, which we had not confirmed. What we can confirm is that Search Console is not yet part of the experiment’s record.

Correction, . The earlier version said Search Console was not yet part of the site’s measurement record. The property is now verified and Search Console is part of the record. The historical canonical checks above still describe what we observed before that verification.

They also do not tell us whether a three-step chain matters. The Google redirect page cited above does not give a hop limit, and we are not going to supply one from elsewhere. So we report the chain as it is and do not call it good or bad. If it is later shortened to a single redirect, that will be a new, dated observation, not a correction to this one.

What we learned

On this site, the canonical tags were already right. The part that failed was an address variant that is easy to forget: www. The host returns the redirect today, and the only change on our record is the DNS record, so the gap we can point to was at the DNS layer. Checking every variant from outside, rather than trusting the configuration screens, is what found it.

Terms used

Canonical URL
The address Google chose as most representative from a set of duplicates.
Duplicate cluster
The group of addresses that show the same content.
301 / 308
Redirect codes Google treats as permanent moves.
Host variant
The same site under a different host name, such as www.

Lesson 6.4

robots.txt: what it controls, and what it does not

Published . Sources checked .

robots.txt is a small text file at the root of a site that tells crawlers which paths they may request. This lesson covers what the file does, how Google reads it, where other crawlers differ, and what it cannot do.

You will be able to
Write and read a robots.txt file, predict how Google will interpret it, and say what it cannot control.
Before this lesson
Lesson 6.1, Crawlability.

Concept

Google describes the purpose plainly: “A robots.txt file is used primarily to manage crawler traffic to your site”. Evidence: Documented by Google

A file is made of groups. Each group starts with one or more user-agent lines and is followed by rules. Google’s specification defines the fields: “user-agent: identifies which crawler the rules apply to. allow: a URL path that may be crawled. disallow: a URL path that may not be crawled. sitemap: the complete URL of a sitemap.” Evidence: Documented by Google

The starting point is permissive: “By default, there are no restrictions for crawling for the designated crawlers.” Evidence: Documented by Google

Why it matters

robots.txt controls requests, not visibility. Three limits follow from Google’s own documentation:

  • It is not a way to hide pages. “Don’t use a robots.txt file as a means to hide your web pages (including PDFs and other text-based formats supported by Google) from Google Search results.”
  • Blocked pages can still be indexed. “A page that’s disallowed in robots.txt can still be indexed if linked to from other sites.”
  • It is a request, not a lock. “The instructions in robots.txt files cannot enforce crawler behavior to your site; it’s up to the crawler to obey them.” Evidence: Documented by Google

How Google reads the file

These rules are Google’s. Other crawlers may read the same file differently.

  • Size: “Google enforces a robots.txt file size limit of 500 kibibytes (KiB).”
  • Conflicts: “In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.”
  • Groups do not merge with *: “User agent specific groups and global groups (*) are not combined.” If you add a group for one named crawler, that crawler follows its own group only, not the rules under *.
  • Crawl-delay: Google lists the fields it supports and notes that “other fields such as crawl-delay aren’t supported”.
  • Caching: “Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn’t possible (for example, due to timeouts or 5xx errors)”. In practice, a change may take time to be reflected, because Google can be using a cached copy.
  • Missing file: “Google’s crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn’t exist.” Evidence: Documented by Google

Where other crawlers differ

One example of a difference: Anthropic says its bots honour robots.txt and that “we support the non-standard Crawl-delay extension”, which Google does not. Evidence: Documented by Anthropic So a Crawl-delay line can matter to one crawler and be ignored by another. Lesson 6.11 covers AI crawlers one by one.

How to do it

  1. Open your file. Go to https://your-site/robots.txt. Google: “You must place the robots.txt file in the top-level directory of a site, on a supported protocol.” Evidence: Documented by Google
  2. Read it group by group. For each user-agent line, list the allow and disallow rules under it.
  3. Check for named groups. If a crawler has its own group, remember it ignores the * group entirely (Google’s rule above).
  4. Check what you are trying to achieve. If the goal is to keep a page out of search results, robots.txt is the wrong tool (see “Why it matters”).
  5. Add your sitemap. Google’s sitemap guide says to “Insert the following line anywhere in your robots.txt file, specifying the path to your sitemap”. Evidence: Documented by Google
  6. Expect a delay after changes. Google may still be using a cached copy: generally up to 24 hours, and longer if it cannot refresh the file.

What we changed on this site

Nothing was changed for this lesson. The file was created on , Day 1 of the experiment, and reads, in full:

User-agent: *
Allow: /

Sitemap: https://learnaiseocourse.com/sitemap.xml

Line by line:

  • User-agent: * starts a group that applies to any crawler without a group of its own. This file has no other groups.
  • Allow: / permits every path. By Google’s rule that there are no restrictions by default, this line restates the default rather than changing anything for Google.
  • Sitemap: ... gives the sitemap’s full URL. It sits outside any group.

What happened

Evidence: Observation

The production file returned HTTP 200 as text/plain, and its bytes matched the file in the site’s repository exactly.

What this does not prove

The file does not block any crawler, so robots.txt is not an observed blocking mechanism on this site. That does not establish that any platform will crawl, index, retrieve, mention or cite the site. Other layers, such as a CDN’s bot rules (lesson 6.1), and each platform’s own choices, are not covered by this check.

What we learned

On this site, robots.txt blocks nothing: it restates Google’s default and points to the sitemap. The file starts to matter once you add rules, because that is when the grouping and conflict rules above decide what actually happens.

Terms used

User-agent token
The name in a user-agent line that a crawler matches against.
Group
One or more user-agent lines and the rules under them.
Allow / Disallow
Rules for which URL paths may or may not be crawled.
Wildcard group (*)
The group for crawlers without a group of their own.

Lesson 6.5

Sitemap: what it does, and whether you need one

Published . Sources checked .

A sitemap is a file that lists the pages you want search engines to know about. This lesson covers what Google says a sitemap does and does not do, how to decide whether you need one, and which fields Google actually uses. It ends with the change we made to this site’s sitemap, and what that change does and does not show.

You will be able to
Decide whether your site needs a sitemap, build one that only contains fields Google uses, and keep its dates accurate.
Before this lesson
Lesson 6.3, Canonicalization.

Concept

Google’s definition: “A sitemap is a file where you provide information about the pages, videos, and other files on your site, and the relationships between them.” Evidence: Documented by Google

What it does, and its limit, in one sentence: “A sitemap helps search engines discover URLs on your site, but it doesn’t guarantee that all the items in your sitemap will be crawled and indexed.” Evidence: Documented by Google

Why it matters, and when it does not

Google lists reasons you might need a sitemap and reasons you might not.

You might need one if (Google’s list): your site is large; “Your site is new and has few external links to it”; or your site has a lot of rich media or appears in Google News.

You might not need one if (Google’s list): “Your site is ‘small’. By small, we mean about 500 pages or fewer on your site”; your site is comprehensively linked internally; and you don’t have many media files or news pages you want in search results. Evidence: Documented by Google

These can point both ways at once. This site is a good example: it is new and has few external links, which is on the “might need” list, and it is small and fully linked from its navigation, which is on the “might not” list. Google’s overall view is that “in most cases, your site will benefit from having a sitemap.” Evidence: Documented by Google We keep one because the site is new and has few links pointing at it.

The fields Google uses

  • <loc> is the page’s address. Use the canonical URL (lesson 6.3).
  • <priority> and <changefreq>: “Google ignores <priority> and <changefreq> values.” Leave them out.
  • <lastmod>: “Google uses the <lastmod> value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.” Evidence: Documented by Google

What counts as a change worth recording: “The <lastmod> value should reflect the date and time of the last significant update to the page. For example, an update to the main content, the structured data, or links on the page is generally considered significant, however an update to the copyright date is not.” Evidence: Documented by Google

Note the condition. Google’s statement is that it uses <lastmod> when it is accurate. It is not a promise that any particular site’s dates will be used.

How to do it

  1. Decide whether you need one, using Google’s two lists above.
  2. List canonical URLs only. One <loc> per page, in the exact form your canonical tags use.
  3. Add <lastmod> only if you can keep it accurate. Google uses the field only when it is accurate, so a date you do not maintain is of no use to it.
  4. Decide what “significant” means for your pages, using Google’s definition, and write it down.
  5. Point to the sitemap. Google says to “Insert the following line anywhere in your robots.txt file, specifying the path to your sitemap” (lesson 6.4). Evidence: Documented by Google
  6. Expect no guarantee. Google calls submitting a sitemap “merely a hint: it doesn’t guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site.” Evidence: Documented by Google

What we changed on this site

Evidence: Observation

Before. The sitemap listed the five pages with <loc> only:

<url>
  <loc>https://learnaiseocourse.com/ai-seo-course</loc>
</url>

The change. We added <lastmod> to all five URLs:

<url>
  <loc>https://learnaiseocourse.com/ai-seo-course</loc>
  <lastmod>2026-09-19</lastmod>
</url>

How the dates were chosen. Each date is the UTC date of the last significant change to that page, taken from the site’s git history. A page is built from more files than its own (shared navigation, the course data, lesson files), so we listed every file that renders each page and checked what each commit actually changed on that page, rather than taking the latest commit that touched any related file.

Page<lastmod>
/2026-09-19
/ai-seo-course2026-09-19
/experiments2026-09-19
/research2026-09-18
/about2026-09-18

Our maintenance rule. A page’s date is updated in the same change that significantly alters that page, and never for unrelated deploys or style-only changes.

One judgment of our own. The homepage and the experiment page show a “Day N of 30” counter that changes every day by itself. We do not treat that as a significant change, by analogy with Google’s copyright-date example. Google’s documentation does not address this case, so this is our interpretation, not Google’s rule.

What happened

Evidence: Observation

The sitemap went live on 19 September. Checked in production at 09:08 UTC: it returned HTTP 200 as application/xml, parsed as valid XML, listed the same five URLs (each returning 200), carried exactly the dates in the table above, had no <priority> or <changefreq>, and matched the file in the site’s repository byte for byte.

What this does not prove

This shows what we changed and that the live file matches what we intended. It does not show that Google uses these dates. The sitemap is referenced in robots.txt. Whether Google uses the dates is a question for later, and only Search Console data could begin to answer it.

Update, . This paragraph first said that the check did not show Google had read the sitemap, and that “a Search Console sitemap submission has not been recorded in this experiment, and Search Console is not yet part of the site’s measurement record”. Both were accurate when published. The sitemap was submitted on 20 September 2026 and Search Console reported it read the same day, status Success, with 5 URLs discovered. Discovered means Google read 5 URLs out of the file. It does not mean any of them are crawled or indexed, and it still does not show whether Google uses the <lastmod> dates.

What we learned

On this site, the hard part of <lastmod> was not adding it. It was deciding what counts as a change for each page, and it will be keeping to that rule afterwards. One commit touched a file that every page imports, but changed only something the other pages never display; counting it would have made two pages look updated when they were not.

Terms used

Sitemap
A file listing the URLs you want search engines to know about.
<loc>
A page’s address in the sitemap.
<lastmod>
The date of the page’s last significant change.
Significant update
Google’s term for a change to main content, structured data, or links on the page.

Lesson 6.6

JavaScript rendering: can crawlers see what your JavaScript builds?

Published . Sources checked .

Many sites build part of their page in the browser with JavaScript. A crawler that downloads the page but does not run that JavaScript sees only what the server sent. This lesson covers what Google documents about rendering, what is not documented for other crawlers, and how to check which of your content is in the page before any JavaScript runs.

You will be able to
Check whether your important content is in the HTML your server sends, and say what is and is not known about how different crawlers handle JavaScript.
Before this lesson
Lesson 6.1, Crawlability.

Concept

When a crawler requests a page, the server returns HTML. If the page’s content is already in that HTML, a crawler that downloads the HTML can receive that content without running JavaScript. If the content is added afterwards by JavaScript running in a browser, only a crawler that runs the JavaScript will see it.

Running the JavaScript and building the final page is called rendering.

What Google documents

Google renders pages: “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome, similar to how your browser renders pages you visit.” Evidence: Documented by Google

Rendering is not instant. “Googlebot queues pages for both crawling and rendering.” “The page may stay on this queue for a few seconds, but it can take longer than that.” Then “a headless Chromium renders the page and executes the JavaScript.” Evidence: Documented by Google

Blocked JavaScript is not run: “Google Search won’t render JavaScript from blocked files or on blocked pages.” Evidence: Documented by Google The same applies to Google’s AI features: “Google is able to process content within JavaScript as long as it isn’t blocked.” Evidence: Documented by Google

Why it matters

Google also says why sending content in the HTML is still worth doing, even though Google itself renders: “server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript.” Evidence: Documented by Google

Notice what that sentence does and does not say. It says not all bots can run JavaScript. It does not say which ones.

What is not documented

We checked the crawler documentation of three AI platforms: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot) and Perplexity (PerplexityBot, Perplexity-User). None of the three pages mentions JavaScript or rendering at all.

Evidence: Observation

So whether those crawlers run JavaScript is unknown from their own documentation. Some SEO guidance states that AI crawlers do not run JavaScript. Their operators’ documentation, as of this check, neither confirms nor denies it.

The practical conclusion does not need that question answered. If your important content is in the HTML the server sends, a crawler can receive it whether or not it runs JavaScript. Google’s sentence above gives the reason to do this; the unknowns about other crawlers are a reason not to depend on rendering.

How to do it

  1. Pick a sentence from the main content of a page, one you care about being found.
  2. Fetch the raw HTML and search for it. This is what a crawler gets before running any JavaScript:
    curl -s https://your-site/your-page | grep -c "your sentence"
    1 or more means the sentence is in the HTML the server sent. 0 means it is not there, or it is split across tags or written with HTML entities (such as &rsquo; for an apostrophe); check with your browser’s “View page source”.
  3. Compare with the rendered page. Open the page in a browser. If the sentence is visible there but missing from the raw HTML, it is added by JavaScript, and only crawlers that render will see it.
  4. Check nothing important is blocked. If JavaScript files are disallowed in robots.txt, Google will not run them (lesson 6.4).

What we changed on this site

Nothing was changed for this lesson. The site has used Next.js with pages generated on the server since it was built. The response headers show pages served from Vercel’s cache of pre-rendered output.

What happened

Evidence: Observation

At 09:18 UTC we fetched /ai-seo-course from production without running any JavaScript, and counted the words inside each published lesson. We then loaded the same page in a browser, let its JavaScript run, and counted again.

LessonWords in raw HTMLWords after JavaScript ran
6.1 Crawlability1,0481,048
6.3 Canonicalization1,0281,028
6.4 robots.txt909909
6.5 Sitemap1,0521,052

The counts match. The lesson text is in the server’s response, and running the page’s JavaScript does not change how much of it there is.

What this does not prove

This shows that the published lesson text is available without running JavaScript. It does not show that any AI crawler has fetched the page, whether any of them runs JavaScript, or that any AI system has retrieved, used or cited the content. Those are separate questions (lesson 6.1).

A word count also cannot catch everything. It shows the same amount of text is present; it does not check every character, or content outside the lessons.

What we learned

The useful question was not “does this crawler run JavaScript?”, which the AI platforms’ documentation does not answer. It was “is our content available without running JavaScript?”, which we could check ourselves. For the published lesson text, the answer is yes.

Terms used

Rendering
Running a page’s JavaScript to build the final page.
Raw HTML
What the server sends, before any JavaScript runs.
Server-side rendering / pre-rendering
Building the page’s HTML on the server, so the content is in the response.
Render queue
Google’s queue of pages waiting to be rendered.

Lesson 6.7

Structured data: what it describes, and what it does not guarantee

Published . Sources checked .

Structured data is code you add to a page to describe what the page is about in a format machines can read. This lesson covers what Google says it uses structured data for, the rules that apply to it, which rich results still exist, and the decision this site made about marking up its own course.

You will be able to
Decide what structured data a page should carry, check that it matches what the page shows, and tell a rich-result opportunity apart from a description.

Concept

Google: “Google uses structured data that it finds on the web to understand the content of the page, as well as to gather information about the web and the world in general”. Evidence: Documented by Google

There are three formats. Google’s advice is to pick “a format that’s easiest for you to implement and maintain (in most cases, that’s JSON-LD); all 3 formats are equally fine for Google”. Evidence: Documented by Google

Why it matters, and its limits

Four rules from Google’s structured data policies and AI guidance:

  • It must describe what readers can see. “Don’t mark up content that is not visible to readers of the page.”
  • It guarantees nothing. “Google does not guarantee that your structured data will show up in search results, even if your page is marked up correctly according to the Rich Results Test.”
  • A penalty affects rich results, not ranking. “A structured data manual action means that a page loses eligibility for appearance as a rich result; it doesn’t affect how the page ranks in Google web search.” Evidence: Documented by Google
  • It is not required for AI features. “Structured data isn’t required for generative AI search, and there’s no special schema.org markup you need to add.” Google still suggests using it, “as it helps with being eligible for rich results on Google Search.” Evidence: Documented by Google

Rich results that no longer exist

Some guides still recommend markup for features Google has retired. Google’s documentation changelog records:

  • How-to: documentation removed in September 2023, “as this rich result is no longer shown in search results, on both desktop and mobile devices.”
  • Course info: documentation removed on 9 September 2025, because “These structured data types are no longer shown in Google Search results.”
  • FAQ: “This feature will no longer appear in Google Search starting May 7, 2026.” Evidence: Documented by Google

One course-related result does still exist: course list. “The course list rich result is available in English in all regions where Google Search is available.” Its conditions include “You must mark up at least three courses”, and its definition of a course includes being “led by one or more instructors with a roster of students.” Evidence: Documented by Google

Check the current documentation before adding markup for a rich result. The list of what exists changes.

How to do it

  1. Start from what the page shows. Only mark up things a reader can see on that page.
  2. Check the feature still exists. Look it up in Google’s current structured data documentation and its changelog, not in an older guide.
  3. Read the eligibility rules, not just the property list. The course list result, for example, has requirements about the number of courses and what counts as a course.
  4. Use JSON-LD unless another format is easier for you to maintain.
  5. Validate with Google’s Rich Results Test, and remember that passing it is not a guarantee of display.

What we changed on this site

Evidence: Observation

What the site carries. Re-checked in production on 20 September: the homepage has one WebSite block, the About page has one Person block for the author, and the other three pages have none. /ai-seo-course has no Course markup.

Why there is no Course markup. This was decided three times, and the reason has stayed the same:

  • 18 September (DEC-005): the course content did not exist yet, so Course markup would have described something not on the page.
  • 18 September (DEC-011): after the page was redesigned, it showed a plan for the course, not the course itself. Same conclusion.
  • 19 September (DEC-013): the trigger for adding it was changed from “a certain number of modules written” to “the page genuinely represents a course”, checked against Google’s documentation at that time.

The reason is the rule above: structured data should describe what the page shows.

What we got wrong on the way. The first draft of DEC-013 said Google no longer had any rich result for courses. That was wrong: the course info result was retired, but the course list result still exists. We corrected the decision the same day with a dated note, rather than rewriting it.

Separately, on rich-result eligibility. This site does not currently meet Google’s documented eligibility requirements for the course list rich result: it has one course rather than the “at least three courses” the documentation requires, and it does not have a roster of students.

The second half of that is our reading, not Google’s words. Google’s definition of a course includes being “led by one or more instructors with a roster of students”. This site has no signup and keeps no list of enrolled people, so we do not believe it meets that part of the definition. Google has not ruled on this site.

Note the word currently. These are facts about the site today, not a permanent bar, and they are separate from the reason Course markup was left out. That reason was always the visible-content rule above.

What happened

Evidence: Observation

The production check on 20 September found exactly the markup the decisions describe: one WebSite block on the homepage, one Person block on the About page, and nothing on the course, experiment and research pages.

What this does not prove

This shows what markup the site sends. It does not show that Google has read it, or used it to understand the site or its author. We are not aware of a rich result this markup would produce, so there is no display to observe either.

What we learned

Most of the work was deciding what not to mark up. The rule that settled it was Google’s plainest one: describe what the page shows. The rich-result rules were useful to know, but they were never the reason.

Terms used

Structured data
Code describing a page’s content in a machine-readable format.
JSON-LD
The structured data format Google says is easiest in most cases.
Rich result
An enhanced search result display that some structured data can make a page eligible for.
Manual action
An action Google takes against a page; for structured data, Google says it removes rich-result eligibility.

Lesson 6.9

Page architecture: URLs, headings and how much to put on one page

Published . Sources checked .

Page architecture is how you divide content between pages, what you call the addresses, and how you organise what is on each one. This lesson covers what Google documents about URLs and page structure, the myths it explicitly warns against, and how this course ended up on a single URL.

You will be able to
Choose URLs a person can read, structure a page with headings that match its content, and decide whether something deserves its own page or belongs as a section of an existing one.

Concept

Google’s guidance on URLs is short and about readability. “Consider organizing your content so that URLs are constructed logically and in a manner that is most intelligible to humans.” Evidence: Documented by Google

And on what to put in them: “When possible, use readable words rather than long ID numbers in your URLs.” Evidence: Documented by Google

There is one specific thing Google says not to do: “Don’t use fragments to change the content of a page, as Google Search generally doesn’t support URL fragments.” Evidence: Documented by Google

Read that carefully, because it is easy to over-apply. Our reading: the instruction is about using a fragment to change what the page contains, for example loading different content at #tab-two than at #tab-one while the address stays the same. It is not an instruction against linking to a heading further down a page whose content is already there. Google’s sentence is quoted in full above so you can weigh that reading yourself.

Why it matters

Google’s own mythbusting

The generative AI guide has a section on overfocusing, and two of its items are about page structure. On chunking: “There’s no requirement to break your content into tiny pieces for AI to better understand it.” Evidence: Documented by Google

On writing style: “You don’t need to write in a specific way just for generative AI search.” Evidence: Documented by Google

A large amount of published AI SEO advice contradicts those two sentences directly, including rules about ideal paragraph lengths and answer blocks of a specific word count. This course previously carried such a rule in its own specification and removed it, because no primary source supports it. It is not taught here.

Splitting a page is a decision with a policy attached

The temptation with any long page is to break it into many small ones, each aimed at a slightly different phrasing of the same question. Google’s spam policies name that pattern: “Scaled content abuse is when many pages are generated for the primary purpose of manipulating search rankings and not helping users.” Evidence: Documented by Google

And on what it usually looks like: “This abusive practice is typically focused on creating large amounts of unoriginal content that provides little to no value to users, no matter how it’s created.” Evidence: Documented by Google

The test in that first sentence is purpose. Splitting a genuinely large topic into pages that each stand on their own is not the same as generating variants to catch phrasings. The difference is not the page count.

How to do it

  1. Write the URL for a person. If someone reads it aloud, it should describe the page. Words, not identifiers.
  2. Keep the address stable once published. If you change it later, you may need redirects and may break existing references. Decide before publishing, not after.
  3. Use headings that describe their sections and keep the hierarchy meaningful. A heading is a promise about what follows.
  4. Do not use a fragment to swap the content. Fragments are for moving to a place in a page whose content is already present.
  5. Before splitting a page, ask what the new page is for. If it exists to catch a phrasing rather than to be read, do not make it.

What we changed on this site

Nothing was changed for this lesson. The architecture it describes was decided earlier and is recorded:

  • One URL for the course. The whole of Module 6 lives at /ai-seo-course, with each lesson at a fragment such as #module-6-internal-links. Every lesson’s text is in that page’s HTML at the same time. The fragment moves the reader’s position. It does not change what the page contains, which is the distinction Google’s sentence draws.
  • Criteria before splitting. Four conditions a module must meet before it earns its own URL: substantial unique content that stands without the parent page, its own first-party evidence, real demand as a separate query, and a split that helps the reader rather than the keyword. None of the nine modules currently meets all four.
  • A stated refusal. Pages will not be created primarily to catch a similar query and funnel readers elsewhere, which is the pattern the spam policy above describes.
  • DEC-014, 19 September. The module list changed after readers could already see it. The change was stated on the page with a date rather than made silently.

What happened

Evidence: Observation

From the live page audit on 20 September:

CheckResult
URL/ai-seo-course, words, no identifiers
<h1> elements1
Elements carrying an id16, all unique
Fragment links resolving to an existing id34 of 34
Content behind a fragmentNone. All lesson text is present in the served HTML
Pages created per query variation0

The last row is a statement about what we have not built. The site publishes five pages, listed in sitemap.xml, and none of them exists to capture a phrasing of the target query.

What this does not prove

None of this shows that the single-page structure ranks better or worse than a split one. We have not run that comparison, and running it properly would mean publishing the same content both ways, which we are not going to do.

It also does not show that Google approves of the structure. It shows the structure follows the documented guidance we could find. Those are different things, and only Google can report the second.

Update, . The audit above, and the “one URL for the course” and “none of the nine modules currently meets all four” statements, describe the architecture as it stood on 20 September and are not restated here. DEC-033 revised the criteria to apply to an individual lesson within an in-progress module, not only to a whole module, and judged two lessons against it: 1.5, Query fan-out and 6.12, llms.txt. Both now have their own URL. The stated refusal above is unchanged: neither page was created to catch a phrasing of the target query, and no other lesson has been split.

What we learned

The useful part of Google’s URL guidance was not the URL. It was the fragment sentence, because it is the one most likely to be misread. Taken as “fragments are bad for SEO”, it would argue against the table of contents on this page. Read as written, it argues against something this page does not do.

The harder decision was page count, and the documentation does not settle it. The spam policy gives a test of purpose, not a number, so the criteria had to be written before the temptation arrived rather than during it.

Terms used

URL fragment
The part of an address after #, naming a position within a page.
Semantic HTML
Elements chosen for what the content is rather than how it looks.
Heading hierarchy
The nesting of h1 to h6 describing a page’s structure.
Scaled content abuse
Google’s term for generating many pages primarily to manipulate rankings rather than to help readers.

Lesson 6.10

Performance: Core Web Vitals, and what they are worth

Published . Sources checked .

Core Web Vitals are three measurements Google publishes thresholds for. This lesson covers what the three are, what Google says about how much they matter for ranking, the difference between measuring in a lab and measuring real visits, and what this site can and cannot currently measure about itself.

You will be able to
Name the three metrics and their thresholds, tell lab data from field data, and say what your own numbers do and do not support.
Before this lesson
Module 0.

Concept

Google publishes three metrics with thresholds.

Largest Contentful Paint. “Measures loading performance. To provide a good user experience, strive to have LCP occur within the first 2.5 seconds of the page starting to load.” Evidence: Documented by Google

Interaction to Next Paint. “Measures responsiveness. To provide a good user experience, strive to have an INP of less than 200 milliseconds.” Evidence: Documented by Google

Cumulative Layout Shift. “Measures visual stability. To provide a good user experience, strive to have a CLS score of less than 0.1.” Evidence: Documented by Google

Note what is not on that list. Time to first byte is not a Core Web Vital. It is a useful number, and it feeds into loading, but it is not one of the three and no threshold above applies to it.

Why it matters, and how much

Google is unusually direct about the weight of this, and the four sentences below are worth reading together rather than picking one.

There is no single switch: “There is no single signal. Our core ranking systems look at a variety of signals that align with overall page experience.” Evidence: Documented by Google

They are used: “Core Web Vitals are used by our ranking systems.” Evidence: Documented by Google

Content can still win: “Google Search always seeks to show the most relevant content, even if the page experience is sub-par.” Evidence: Documented by Google

And the sentence most performance advice leaves out: “Keep in mind that getting good results in reports like Search Console’s Core Web Vitals report or third-party tools doesn’t guarantee that your pages will rank at the top of Google Search results; there’s more to great page experience than Core Web Vitals scores alone. These scores are meant to help you to improve your site for your users overall, and trying to get a perfect score just for SEO reasons may not be the best use of your time.” Evidence: Documented by Google

That is Google saying a perfect score chased for ranking reasons may be wasted effort. This lesson does not claim a perfect score improves rankings.

Lab and field are not the same number

A lab measurement is one run, on one machine, on one connection, usually with nothing else happening. A field measurement is what real visitors actually experienced, collected from real sessions.

They answer different questions. Lab data tells you what the page can do under controlled conditions and is useful while you are changing things, because you can run it again immediately. Field data tells you what real visitors experienced, which is different from a controlled lab measurement. You cannot have it until real people have visited.

A new site has no field data because it has no visitors yet. That is this site’s position, and no amount of local measurement substitutes for it.

How to do it

  1. Know which of the three you are looking at. LCP, INP, CLS. If a tool gives you one number for “performance”, that is not one of them.
  2. Measure in a lab while you work. Repeatable, immediate, good for catching a regression you just introduced.
  3. Get field data once you have visitors. Search Console’s Core Web Vitals report and other field sources report real sessions.
  4. Record the method with the number. A response time from your own laptop on home broadband is not the same measurement as a server-side timing, and writing down which one you ran is what makes it comparable later.
  5. Stop before the last few points. Google’s own sentence above says chasing a perfect score for SEO may not be the best use of your time.

What we changed on this site

Nothing was changed for this lesson. The measurements below were taken to record a starting position, not to test a change.

What happened

Evidence: Observation

MeasurementValueMethod
Field LCP, INP, CLSNot recordedNo field data exists yet
Full response time, /ai-seo-course0.36 s to 1.55 s across 6 samplesCache-busted requests from one machine
Served HTML101,027 bytesSame requests
Script files requested13Parsed from the served HTML
Total JavaScript, uncompressedapprox. 523 KBSum of decompressed bodies
Total JavaScript, compressedNot reliably measuredSee below

Three limits on the table above, stated rather than buried.

The response times are not server TTFB. They are the full time for a request to complete from one laptop on one connection, which includes the network path and the client. A server-side TTFB would be a smaller and different number.

The compressed JavaScript figure is missing because the tool used decompresses responses automatically, so the sizes it reported were decompressed ones. Reporting that 523 KB as a transfer size would have been wrong, so it is labelled uncompressed and the compressed figure is left unrecorded.

These numbers are also not comparable to the earlier figures recorded on 19 September, which used a different method. Two numbers produced by different methods should not be put in the same trend line.

Page weight over the day

Raw HTML for /ai-seo-course, recorded as lessons were published on 20 September:

PointBytesPublished lessons
Start of day72,3195
After corrections73,6585
After lesson 6.289,9406
After lesson 6.7100,9977
After lessons 6.8 to 6.12158,76512

This series is descriptive, not a performance budget. The page became substantially larger as lessons were added. No ranking or user-experience effect is inferred from these measurements, and none has been measured.

Five points in one day is a record, not a trend.

What this does not prove

None of these numbers is a Core Web Vital. LCP, INP and CLS are not in the table because none of them has been measured on this site in a way worth recording, and field versions of them do not exist yet.

The response times do not show what a visitor elsewhere would experience. Six samples from one machine on one connection on one afternoon is a sample of that machine, that connection and that afternoon.

Nothing here shows any effect on ranking. No change was made, so there is no before and after to compare.

What we learned

The measurement that mattered most was the one we could not make. Field Core Web Vitals need real visitors, and a new site does not have them, so the row says Not recorded and will stay that way until it can say something else.

The second lesson was about method. A convenient tool produced a JavaScript size that looked precise and was measuring the wrong thing. Keeping the method next to the number is what caught it.

Terms used

LCP
Largest Contentful Paint. Loading, threshold within 2.5 seconds.
INP
Interaction to Next Paint. Responsiveness, threshold under 200 milliseconds.
CLS
Cumulative Layout Shift. Visual stability, threshold under 0.1.
Lab data
Measured in controlled conditions, repeatable on demand.
Field data
Collected from real visits. Cannot be simulated.
TTFB
Time to first byte. Useful, but not a Core Web Vital.

Lesson 6.11

AI crawler access: which agents exist, and what blocking each one does

Published . Sources checked .

Different AI companies send different agents, for different purposes, and blocking one does not do what blocking another does. This lesson covers what each operator documents about its own agents, one Google control that is routinely described wrongly, and what a site owner can actually observe.

You will be able to
Name the documented agents for the major platforms, say what blocking each is documented to change, and tell the difference between a request arriving and a page being used in an answer.
Before this lesson
Lesson 6.4, robots.txt.

Concept

Three kinds of agent get mixed together in most discussions, and the distinction comes from the operators’ own descriptions:

  • A training crawler collects content that may be used to train models.
  • A search crawler builds an index so a platform can surface pages.
  • A user-triggered fetcher goes and gets a page because a person just asked a question that needs it.

They are not interchangeable, and robots.txt does not affect all three the same way.

Google-Extended is not a crawler

This is the single most common error in AI crawler advice, so it gets its own section.

Google’s documentation: “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.” Evidence: Documented by Google

So there is no Google-Extended bot. Nothing arrives at your server identifying itself that way. It is a token you write in robots.txt to control how content that Google has already crawled may be used.

What it controls: content “may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding ... in Gemini Apps and Grounding with Google Search on Vertex AI”. Evidence: Documented by Google

And what it does not do: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Evidence: Documented by Google

Blocking Google-Extended is therefore not a way to stay out of AI Overviews. For Google Search AI features the relevant controls are ordinary Googlebot rules, the snippet controls, and the Search Console generative AI control covered in lesson 6.2.

What each operator documents

Every line below comes from the operator’s own page, re-read on 20 September 2026.

OpenAI

“OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features.” Evidence: Documented by OpenAI

“GPTBot is used to crawl content that may be used in training our generative AI foundation models.” Disallowing it “indicates a site’s content should not be used in training generative AI foundation models”. Evidence: Documented by OpenAI

“When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.” And the part that matters for robots.txt: “Because these actions are initiated by a user, robots.txt rules may not apply.” Evidence: Documented by OpenAI

The three are separate switches: “Each setting is independent of the others.” So a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to stay out of training. Evidence: Documented by OpenAI

Anthropic

“ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” Evidence: Documented by Anthropic

“Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.” Evidence: Documented by Anthropic

“Claude-SearchBot navigates the web to improve search result quality for users.” Evidence: Documented by Anthropic

On robots.txt: “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” Anthropic also states: “we support the non-standard Crawl-delay extension to robots.txt”, which lesson 6.4 notes Google does not. Evidence: Documented by Anthropic

Perplexity

Of PerplexityBot: “It is not used to crawl content for AI foundation models.” Evidence: Documented by Perplexity

Of Perplexity-User: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” It is also “not used for web crawling or to collect content for training AI foundation models”. Evidence: Documented by Perplexity

One deliberate gap. On the page re-read on 20 September 2026, we did not find a sentence stating whether PerplexityBot obeys robots.txt. The page recommends allowing it. It does not say what happens if you disallow it. This lesson therefore does not claim PerplexityBot obeys robots.txt, and you should not assume it from the fact that other operators say so about their own agents.

Summary
TokenOperatorDocumented purposerobots.txt
Google-ExtendedGoogleControl token for Gemini training and groundingControl token, not an agent
OAI-SearchBotOpenAISurface sites in ChatGPT searchObeys
GPTBotOpenAITraining foundation modelsObeys
ChatGPT-UserOpenAIFetches for a user’s question“may not apply”
ClaudeBotAnthropicCollects content for trainingHonours robots.txt
Claude-SearchBotAnthropicSearch result qualityHonours robots.txt
Claude-UserAnthropicFetches for a user’s questionHonours robots.txt
PerplexityBotPerplexityNot used for foundation-model trainingNot stated on the page
Perplexity-UserPerplexityFetches for a user’s question“generally ignores”

How to do it

  1. Decide per purpose, not per company. Training, search and user-triggered fetching are different questions. Most operators let you answer them separately.
  2. Read the operator’s own page. These pages change, and no third-party summary is a substitute, including this one. Check the date.
  3. Remember robots.txt is a request. Lesson 6.4 covers this. Two operators state outright that user-triggered fetches may not follow it.
  4. Do not use Google-Extended for Search AI features. It does not control them.
  5. Decide what you would do with the answer before building measurement. Counting requests is not the same as learning whether you were cited.

What we changed on this site

Nothing was changed for this lesson. The relevant configuration is unchanged since 18 September:

robots.txt is User-agent: * and Allow: /, plus the sitemap line. No agent is disallowed. Nothing in that file distinguishes between training, search and user-triggered fetching, because it makes no distinctions at all.

What happened

Evidence: Observation

The table above was verified against each operator’s page on the drafting date. That is the only first-party work this lesson reports, and it is a check of documentation, not of traffic.

What we cannot measure, and why

This is the important part of the lesson, and it is a negative result.

Requests are not recorded in a usable way. Checked on 19 September: this site runs on Vercel’s Hobby plan, which retains runtime logs for one hour. A 30-day study cannot be built on a one-hour window.

Cloudflare’s free tier identifies by user-agent string. Cloudflare’s own documentation says its reports identify AI crawlers “based on their user agent strings”, with more thorough detection on paid plans. A user-agent string is text the requester sends. It is not proof of who sent it. Evidence: Documented by Cloudflare

We contaminated one day ourselves. On 19 September we sent requests carrying ten AI crawler user-agent strings from our own machine, in a window recorded to the second. Any user-agent-based report would have counted those as AI crawler traffic. That date is excluded from crawler evidence, and spoofed checks are not to be repeated during the measurement window.

A request is not a citation. Even a genuine fetch from a user-triggered agent shows that a page was retrieved, not that it was used in an answer, and not why. The ladder in lesson 6.2 applies here too.

So: retrieval by AI platforms is not observable in this experiment with the current stack. That is recorded as a limitation rather than worked around, and no crawler-visit figures will be published for this site unless a reliable record exists.

What we learned

The most useful thing in the operators’ documentation was the seams: the places where one of them declines to say something. Perplexity does not say PerplexityBot obeys robots.txt. OpenAI says robots.txt “may not apply” to user-triggered fetches. Those gaps are the actual state of the art, and smoothing them into a tidy table of “all bots obey robots.txt” would have been wrong in both directions.

The second thing is that we cannot measure what we most wanted to measure. Writing that down is the result.

Terms used

Training crawler
Collects content that may be used to train models.
Search crawler
Builds an index so a platform can surface pages.
User-triggered fetcher
Retrieves a page because a person just asked something.
Product token
A name used in robots.txt to control a behaviour, with no user agent of its own. Google-Extended is one.
User agent string
Text a requester sends to identify itself. Not proof of identity.

Module 7: AI SEO Measurement

How to tell whether any of it worked, and what each number can honestly support. Lessons are published one at a time; 7 of 7 so far.

Lesson 7.1

Setting up Search Console from zero

Published . Sources checked .

Search Console is Google’s own report on how your site appears in Google Search, and it is the only first-party source for that data. One practical limit shapes every decision below: you cannot recover performance data from before the property was verified. On this site that turned a setup decision into a deadline.

Two kinds of property

A Domain property is “a domain-level property that Includes all subdomains (m, www, and so on) and multiple protocols (http, https, ftp)”, and it takes “DNS record verification only”. A URL-prefix property “specifies the start of a URL. Any URLs that begin with this prefix are included in that property.” Evidence: Documented by Google

So a URL-prefix property for https://example.com/ does not cover http://example.com/ and does not cover a subdomain. What this costs you is not the setting but the history: choosing a wider property later does not recover the period you did not measure.

Verification, and how it lapses

“HTML file upload can be used for URL-prefix properties, but not Domain properties.” The same restriction applies to the HTML tag, Google Analytics and Tag Manager methods, which leaves a Domain property one route: a DNS record. Evidence: Documented by Google

Verification is not permanent. “Search Console periodically checks if your verification token is still present and valid”, and if it cannot be confirmed, “your permissions on that property will expire after a certain grace period”. On a site that redeploys constantly, keep the verification file in version control, so a deploy restores it rather than removes it. Evidence: Documented by Google

What submitting a sitemap establishes

“Submitting a sitemap means telling Google where to find the file on your site.” Status Success means “the sitemap was fetched and read without any errors”, and discovered pages are “the number of page URLs that were parsed from the sitemap”. Parsed, not crawled and not indexed: “There is no guarantee that a page URL discovered in a sitemap has been or will be crawled or indexed by Google.” Evidence: Documented by Google

See the sitemap lesson for building the file itself, and the indexing lesson for where indexing status actually lives.

Inspecting a URL

URL Inspection is misread more than any other tool here, because by default it is not looking at your live page. “This is not a live test. The results shown are from most recently indexed version of a page, not the live version on the web.” And the green result is weaker than it reads: “‘URL is on Google’ doesn’t actually guarantee that your page will appear in Search results.” Evidence: Documented by Google

Asking again does not help. “There’s a quota for submitting individual URLs and requesting a recrawl multiple times for the same URL won’t get it crawled any faster.” Evidence: Documented by Google

Then you wait. “It can take up to a week to generate data for a newly created site, or a site newly added to Search Console.” Evidence: Documented by Google

How to do it

  1. Choose the property type first. Prefer a Domain property if you can add a DNS record.
  2. If you cannot, check what a URL-prefix property would miss: list the http, https, www and subdomain variants and see what each actually returns.
  3. Verify, and keep the token where a deploy cannot remove it.
  4. Submit your sitemap, and read the status as parsed, not indexed.
  5. Inspect your most important URL, noting whether you are reading the indexed version or a live test.
  6. Write down the verification date. Everything you later claim about impressions or position starts there.

What we did on this site

Evidence: Observation This site’s first repository commit was on 17 September 2026, and the earliest verified production check in this project’s records is 19 September 2026. No first production deployment date has been verified, so none is claimed. The Search Console property was verified on 20 September 2026: a URL-prefix property for https://learnaiseocourse.com/, using the HTML file method.

A Domain property was the better option and we did not take it. The reason recorded at the time was that a DNS record was not available to us, and a 30 day experiment cannot spend three of those days waiting for access before any data starts accumulating. Before settling we measured rather than assumed: http://, http://www and https://www all return 308 redirects to the apex, and the apex canonical is https://learnaiseocourse.com/, so the URL-prefix property covers the URLs Google indexes here. That is a fact about this site’s redirects, not a general rule.

The same day, we submitted /sitemap.xml. It was read that day with status Success, 5 pages discovered and 0 videos. We also confirmed that the Search generative AI control reads Include, which is its default, and changed nothing.

On 22 September 2026 we decided to add a Domain property alongside the existing one rather than replacing it, so the data series already running is not broken. At the time of writing it is decided and not yet created.

What this does not prove

Five discovered URLs is five URLs parsed out of a file. It is not five pages crawled and not five pages indexed. Verification proves ownership; it does not, by itself, establish that a page has been crawled, indexed, or will appear in Search.

Lesson 7.2

Reading each metric, and how each one misleads

Published . Sources checked .

Search Console gives you four numbers. Each is precisely defined, and each is routinely read as something slightly different from what it measures.

Google’s definitions: an impression is “how often someone saw a link to your site on Google”; a click, “how often someone clicked a link from Google to your site”; CTR, “the calculation of (clicks ÷ impressions)”; and average position, “a relative ranking of the position of your link on Google, where 1 is the topmost position”. Evidence: Documented by Google

Impressions: seen is not read

“In general, an impression is counted whenever an item appears in the current page of results, whether or not the item is scrolled into view”, with documented exceptions for carousels and infinite scrolling formats. So a result nobody scrolled to still counts. Read impressions as reach, not attention. Evidence: Documented by Google

Position: the metric Google itself warns about

“The position value is a complex metric that can be misleading if you don’t understand the subtleties.” Evidence: Documented by Google One subtlety is that there is no single results page to be positioned on: “Search results are specific to the time, place, device, and recent history of the person searching.” Evidence: Documented by Google

An average also hides its distribution. The same average position can come from steady mid-page placement or from a mix of very high and very low placements on different queries, and the number does not distinguish them.

Aggregation changes every other number

Aggregated by property, “all results for the same query that point to the same Search Console property are counted once in total”. Aggregated by page, “each unique URL in a Search result is counted separately”. Google states the consequence directly: CTR and average position “are generally higher when aggregating by property, if multiple pages from the same site appear in the Search results”. Evidence: Documented by Google

Google’s own example shows the size of it. One query, three URLs from the same site, one click: property-level reports 100% CTR at position 1, page-level reports 33% CTR at average position 2. Same click, two honest readings, four different numbers. Check the aggregation before comparing any CTR or position figure to anything. Evidence: Documented by Google

What is missing from the table

“Some queries are omitted from the report to protect user privacy. These are called anonymized queries.” They still count towards chart totals unless you filter by query, so the visible rows can add up to less than the total above them. The table is also truncated: “Due to internal limitations, Search Console stores and shows only the most important data rows.” Evidence: Documented by Google

Freshness matters too. “Collected data is usually available in 2 to 3 days”, and the report labels daily data in Pacific Time. Do not read the last two days as final, and check the time zone before comparing against another tool. Evidence: Documented by Google

How to read the report without overstating it

  1. Check the aggregation setting before reading CTR or position.
  2. Read impressions as reach, not attention.
  3. Treat average position as a distribution you cannot see.
  4. Ignore the last two days when judging a direction.
  5. Note what is absent by design: anonymized queries and truncated rows.
  6. Name the window and the filters whenever you quote a number.

The rule we set before we had any data

Evidence: Observation This project fixed an editorial rule on 22 September 2026 while the property was still producing almost nothing, so the rule could not be bent to fit a result. A raw observation may be published as soon as it is read, provided it carries its date, the property, the date range and the filters. A trend claim, meaning any statement of direction, requires at least seven days of data; fourteen or more is preferred before movement is called meaningful; and the final assessment of this site’s first experiment uses thirty days.

This is our editorial rule, not a statistical threshold and not a Google recommendation. Seven days covers one weekly cycle of search behaviour and absorbs the documented two to three day reporting lag. Any lesson here that states a direction names the window it used, and no figure from this site’s Performance report appears above, because none has yet been read into the project’s records with a date range attached.

Lesson 7.3

The Generative AI performance report

Published . Sources checked .

For a while the honest answer to “can I see my AI Overview visibility in Search Console?” was no. That answer is out of date, and it is still widely repeated. There is a Generative AI performance report. It is narrower than most people hope.

What it covers

The report shows “how your site performs in generative AI features on Google Search”, covering AI Overviews and AI Mode and excluding Search Labs experiments. Google says “as of August 31, 2026, we’ve rolled out these insights to all websites worldwide”, and, on the same page, “not all properties have access to the report, as we’re rolling out over time”. Both sentences are on the page. We are not going to reconcile them for you, because reconciling them would mean inventing something Google has not published. Evidence: Documented by Google

Impressions only

The report documents impressions, “how many times links to your site were shown to a user in a generative AI feature on Google Search”, across pages, countries, devices and dates. No clicks, no queries, no position. A site that has not received enough impressions in these features may see no data at all, so zero rows on a small site is ambiguous between no impressions, too few to report, and no access yet. Evidence: Documented by Google

These appearances are not sealed off from the ordinary report. Traffic from AI features is reported in the standard Performance report under the “Web” search type. Evidence: Documented by Google

Two counting rules matter when reading it. “An AI Overview occupies a single position in search results, and all links in the AI Overview are assigned that same position.” And in AI Mode, “if a user asks a follow-up question within AI Mode, they are essentially performing a new query”, with the new response’s data counted against that new query. A conversation of five follow-ups is five queries in your data. Evidence: Documented by Google

The control that decides whether you are included

Before the report there is a setting, under Settings, that manages “your site’s inclusion in the following generative AI features on Google Search”: AI Overviews, AI Mode and Google Discover generative features. Inclusion “is the default control for all properties”, it reached all websites worldwide on 31 August 2026, and exclusion is not instant: “content will be excluded within 1-2 days after the control goes live, but some content may take longer”. Evidence: Documented by Google

The sentence people actually want: this control “only affects whether your content can appear in certain Search generative AI features; this control isn’t used as a ranking or inclusion signal affecting other parts of Search”. So on Google’s own account, the setting governs one surface and is not used as a signal elsewhere. Evidence: Documented by Google

How to use it

  1. Check the control first, and record the date you checked.
  2. Read the report as what it is: impressions, four dimensions.
  3. Treat zero rows on a small site as ambiguous, not as absence.
  4. Cross-read the standard report under the “Web” search type.
  5. Do not convert impressions into clicks or CTR. The report does not publish the numbers that would make that arithmetic valid.

What we found on this site

Evidence: Observation On 20 September 2026 we confirmed that this site’s Search generative AI control reads Include. It is the inherited default and we changed nothing. Recording it mattered more than changing it: an untouched default is still a state, and an experiment that never checked its own eligibility setting would have a hole in it.

Access to the Generative AI performance report for this property has not been confirmed, and no report data has been read into this project’s records. The control being set to Include establishes eligibility to appear, not appearance. When an impression does arrive it will establish that a link was shown in an AI Overview or AI Mode, and nothing about whether anyone read, clicked or was influenced by it.

Lesson 7.4

AI referral traffic and attribution

Published . Sources checked .

When somebody reads about your site inside an AI assistant and clicks through, where does that visit appear? The answer depends on which surface it came from, whether the referrer survived, and which tool you run. This lesson has two halves, deliberately separate: what GA4 can measure, and what this experiment actually measures.

What GA4 can measure

GA4 has a dedicated channel with an exact rule: “the medium exactly matches ‘ai-assistant’. The medium is set to ‘ai-assistant’ and the campaign is set to ‘(ai-assistant)’ if the referrer matches a list of AI Assistants.” Evidence: Documented by Google It was announced on 13 May 2026. Evidence: Documented by Google

Two Google pages give two different example lists. The channel definition names ChatGPT, Gemini, Deepseek, Copilot and Grok; the announcement entry names ChatGPT, Gemini and Claude. Neither is presented as complete, and the matching list itself is not published, so you can confirm that an assistant appears in the channel by observing it, but you cannot work out from the documentation which ones are missing. Evidence: Documented by Google

The exclusion that catches people out

The AI Assistant channel “excludes Google’s AI Overviews and AI Mode”. Those clicks are Organic Search, which “is the channel by which users arrive at your site/app via non-ad links in organic-search results, including Google’s AI Overviews and AI Mode”. Evidence: Documented by Google

So AI traffic is split across two channels by design. A dashboard that treats the AI Assistant channel as “all AI traffic” reports zero for Google’s own AI features no matter how visible you become in them.

Some of it identifies itself as nothing at all. Google’s documented causes for (direct) / (none) include links without campaign parameters, redirects and shorteners that “can strip UTM parameters from the URL”, people typing the address, and ad blockers that “interfere with tracking cookies used to identify the traffic origin”. Evidence: Documented by Google Nothing in that list is specific to AI, which is the point: a stripped referrer is indistinguishable from someone typing your domain from memory. Any AI referral count is therefore a floor, not a total.

What this experiment measures

Evidence: Observation This site does not run GA4. There is no GA4 property, and everything above is documentation we are teaching rather than data we hold. The site runs Vercel Web Analytics on the Hobby plan, which filters by routes, pages, hostname, referrers, country, browsers, devices and operating system. The Referrers panel “only shows top level domains” by default, with a drill-down to the referring pages. Evidence: Documented by Vercel

The limit that shapes this experiment: UTM parameter filtering is “available with Web Analytics Plus and Enterprise”, and the plan table shows it unavailable on Hobby. Evidence: Documented by Vercel So this site can see that a visit arrived from a given referring domain, and cannot break traffic down by campaign parameter at all. That is a real hole, stated rather than worked around.

There is a widely repeated claim that one assistant appends a specific tracking parameter to outbound links. The primary page that would support it returned an HTTP 403 to us on 19 September and again on 22 September 2026, so we have not read it, and it is absent from this lesson rather than cited from somebody’s summary. If you can open it, read it yourself and check the behaviour against your own referrer data.

How to do it

  1. Decide which question you are asking. Traffic from AI assistants and traffic from Google’s AI features are two questions.
  2. Do not build an “AI traffic” total out of the AI Assistant channel alone.
  3. Look at referring domains, available on almost every plan.
  4. Write down what your plan cannot see, as you would record units.
  5. Report a floor as a floor.

No referral figure from this site appears above, because none has been read into the project’s records. When one is, it will establish that a visit arrived carrying a given referrer, and not the share of visits influenced by an AI answer. The visits that lost their referrer are invisible and uncountable.

Lesson 7.5

Measuring AI visibility by sampling answers

Published . Sources checked .

There is no Search Console for ChatGPT. If you want to know whether AI assistants mention your site, somebody has to go and look, and the looking has to survive the fact that the answers change every time.

Why one run is not a measurement

Google says AI Overviews “are only shown when our systems determine that it is additive to classic Search, and as such, often don’t trigger”, so the feature may simply not appear for reasons unrelated to your site. Evidence: Documented by Google AI Mode personalization “references previous searches and activity saved in your Search Services History”, which makes your own account part of the instrument. Evidence: Documented by Google

OpenAI documents that “Chat Completions are non-deterministic by default (which means model outputs may differ from request to request)”. Evidence: Documented by OpenAI Anthropic documents that “even with temperature set to 0, the results will not be fully deterministic and identical inputs may produce different outputs across API calls”. Evidence: Documented by Anthropic

Both of those document the APIs, not the consumer apps most people actually check. We have found no equivalent documentation for the consumer products, so the honest statement is narrower than the conclusion usually drawn from it. It is enough to justify sampling. It is not enough to claim the consumer apps are documented as variable.

Our protocol, and why it is ours

Evidence: First-party experiment The next part is this project’s method, not a platform recommendation and not prescribed by any documentation. Run every prompt three to five times per platform on the same day. Record the result as a rate with its sample size, written cited 3/5, n=5, never as a yes or a no. Compare rates across dates, never individual runs. Record each visibility state separately. Fix the prompt set and the platform list before the first run and do not change them because of a result.

Three to five runs is a practical floor, chosen because a one-person experiment can sustain it daily. It is not a statistically derived sample size. It does not support a significance test, a confidence interval, or a claim that a change from 2/5 to 4/5 is real. If you need those, you need a sample size calculation and far more runs than a person can do by hand, and you should say so rather than dressing a small sample in statistical language.

Record for every run: the prompt wording verbatim, the platform, the date and run window, the account and login state, the country or location context where visible, the interface state, whether an AI answer appeared at all, whether your domain was mentioned, whether a URL was cited, the exact URL when cited, and n. If a platform produces no AI answer, or the prompt does not apply to that interface, record it as not applicable. Never substitute a different query to fill a cell: a substituted query is a different measurement wearing the same label.

What each observation actually establishes

What each signal supports, and what it does not. The rows rest on their own sources; the table is our synthesis.
What you observeWhat it establishesWhat it does not prove
A request with an AI platform’s user agentA request matching that user agent reached your infrastructureThat the operator’s bot sent it. A user agent is a claim, not an identity
A request from a user-triggered fetcherA fetch may have occurred on someone’s behalfThat the page appeared in, or was used for, any answer
An impression in the Generative AI performance reportA link was shown in an AI Overview or AI ModeThat anyone read or clicked it
A referral visit carrying an assistant’s referrerA visit arrived with that referrerThe share of AI-driven visits. Stripped referrers are invisible
A sampled answer citing your URLYour URL was cited in that sampled answerWhy it was selected, or how often it happens in general
A sampled answer naming your site with no linkYour site was mentioned in that answerThat it was cited, or recommended
Retrieval into an answerNothing. It is not observable with this stack

That last row is the one to sit with. A crawler request tells you a page was fetched. It does not tell you the page was used to build an answer. Those are different events, and nothing in this project’s toolkit connects them. See the AI crawler access lesson for what a crawler hit does and does not show.

Mentioned, cited, recommended

These are separate states, not a sequence. A brand can be mentioned without being cited, because a name is not a link, and cited without being recommended, because an answer can use you as a source while advising the reader to do something else. You will see this drawn as a funnel with arrows. Treat the arrows with suspicion: they assert a progression that would have to be demonstrated. Record the states separately and let the relationship between them stay a question.

Microsoft publishes a related metric and is explicit about its limits. Citation Share “is calculated as the percentage of citations attributed to your site out of all citations shown across all sites for that same grounding query”, and is “designed as an observational metric, not a ranking system or a competitive scoreboard. It does not expose competitor domains, represent traffic share, or assign quality scores to content.” Evidence: Documented by Microsoft

Our instrument, fixed before the first run

Evidence: Observation On 22 September 2026, before any run had been performed and therefore before any result could influence the choice, we fixed eight prompts: AI SEO course, best AI SEO course, free AI SEO course, AI SEO course for beginners, how to learn AI SEO, AI SEO training, AI SEO certification course and AI SEO vs traditional SEO. Five platforms: Google AI Overviews, Google AI Mode, ChatGPT Search, Claude web search and Perplexity.

No sampling run has been performed at the time of writing. When the baseline exists it will be published with every field above, including the empty cells. A protocol chosen after seeing the first interesting result is not a protocol: if you pick the prompts once you know which ones mention you, the resulting number measures your selection process rather than your visibility.

Lesson 7.6

Sampling, lag and attribution limits

Published . Sources checked .

Every measurement tool in this course is partial in a way it documents and in a way most people never read. Collecting those limits in one place is what lets you tell a broken measurement from two tools honestly counting different things.

Lag and sampling

Search Console data is “usually available in 2 to 3 days”, and there “can be a lag between when the numbers are calculated and when they are visible”. Evidence: Documented by Google A new property waits longer for anything at all: “it can take up to a week to generate data for a newly created site”. Evidence: Documented by Google

Search Console is also not a ledger. “Most Search Console reports only cover a representative sample of URLs to assess quality of data, rather than a comprehensive listing of all known URLs”, with the Page indexing and Video pages reports as exceptions, still capped at 1,000 displayed URLs. On a five page site this changes nothing. On a large one, a URL absent from a report is not evidence that the URL is absent from Google. Evidence: Documented by Google

Retention, and no way back

“Search Console keeps data for the last 16 months.” Evidence: Documented by Google Bulk export to BigQuery escapes that ceiling going forward, since “data will be accumulated forever for your project, unless you set an expiration time for your data”. It does not reach backwards: “the first export will happen up to 48 hours after your successful configuration in Search Console. The first export includes data for the day of the export.” Evidence: Documented by Google

So the export starts the day you configure it. Like verification itself, its value is decided by how early you did it, and no later effort recovers the difference.

Attribution: two tools, two numbers, both correct

Google publishes why Search Console and Analytics disagree: implementation differences, consent and cookie opt-outs, time zones, attribution models, canonical URLs versus any URL carrying the tracking code, different traffic breakdowns, non-HTML files, and bot filtering. The conclusion to take from it: “since the systems are different, it’s normal and expected that the numbers will be a little different”. Evidence: Documented by Google

GA4 adds a limit of its own. Data thresholds are “applied to prevent anyone viewing a report or exploration from inferring the identity or sensitive information of individual users”, they are system defined, and they cannot be adjusted. A hidden row looks exactly like a row that does not exist. Evidence: Documented by Google

The limits of this site’s own stack

Evidence: Observation Recorded 22 September 2026.

What each tool on this site gives us, and what it cannot.
ToolWhat it gives usLimit
Search Console, PerformanceImpressions, clicks, CTR, position, from 20 September 20262 to 3 day lag; representative sample; 16 month retention
Search Console, Generative AI reportAI Overview and AI Mode impressions, once availableImpressions only. Access for this property unconfirmed
Vercel Web Analytics, HobbyPage views and referring domains50,000 events a month; one month reporting window; no UTM breakdown
Google Analytics 4Nothing. Not installedNot applicable
Server-side crawler logNothing retainedNo capture in place
Retrieval into an AI answerNothingNot observable with this stack

Vercel’s Hobby limits come from its own documentation: 50,000 included events a month, a one month reporting window, and no UTM parameters. Evidence: Documented by Vercel

A conflict inside one page

Vercel’s pricing page says two different things about exceeding the Hobby allowance. In one place, “once you exceed the limits, a three day grace period will start before Vercel will stop capturing events”, after which one option is to “wait 7 days before Vercel will start collecting events again”. In its FAQ, “once you reach the monthly limit, collection pauses until the next billing cycle”. Evidence: Documented by Vercel

A seven day pause and a pause until the next billing cycle are not the same thing. We have not tested which applies, because this site has not approached the limit, so the resolution is recorded as unknown rather than guessed. Documentation contradicts itself more often than people admit, and the useful response is to record the contradiction and mark what you have not tested.

Limits on the AI side

Microsoft’s AI Performance report shows citations displayed as sources in AI-generated answers, and its grounding queries are “a sample of overall citation activity”. Evidence: Documented by Microsoft A sample of citations is not a count of citations, and neither is traffic.

Our own editorial limit

Evidence: Observation One row in this project’s limits is not a tool’s but ours. A dated raw figure may be published as soon as it is read. A statement of direction requires seven days of data at minimum, fourteen or more before movement is called meaningful, and thirty for the final assessment of this site’s first experiment. It sits here deliberately, next to the documented limits and plainly marked as different from them.

Lesson 7.7

Designing an experiment that can fail

Published . Sources checked .

Most published SEO experiments cannot fail. They describe a change, then a rise, and the rise is the conclusion. Nothing was written down that would have counted as the change not working, so nothing could have.

What Google says about timing

“Some changes might take effect in a few hours, others could take several months”, and “in general, you likely want to wait a few weeks to assess whether your work had beneficial effects in Google Search results”. The sentence worth pinning above the desk: “keep in mind that not all changes you make to your website will result in noticeable impact in search results”. Evidence: Documented by Google

Crawling adds its own delay: “crawling can take anywhere from a few days to a few weeks”, and “requesting a crawl does not guarantee that inclusion in search results will happen instantly or even at all”. Evidence: Documented by Google So a test shorter than a few weeks is usually measuring noise, and a null result is a documented normal outcome rather than a sign you did the work badly.

What Google says about running tests

“Don’t show one set of URLs to Googlebot, and a different set to humans. This is called cloaking, and is against our spam policies.” For a test that redirects, “use a 302 (temporary) redirect, not a 301 (permanent) redirect”. And on duration, “run the experiment only as long as necessary”, because “if we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines and take action accordingly”. Evidence: Documented by Google

There is a tension worth naming: the timing guidance pushes tests longer and the testing guidance pushes them shorter. The testing rules govern tests that serve different content to different users, not a change you simply make and then observe. Know which kind you are running.

The nine fields

This form is ours, not Google’s. It is ordinary experimental practice applied to a search change, and its only real virtue is that every field is uncomfortable to fill in afterwards. Write them down before the observation window opens.

  1. Hypothesis, stated so that a specific result would disprove it.
  2. Primary measurement. One metric, named with the tool behind it.
  3. Secondary measurements.
  4. Observation window, with a start date and an end date.
  5. Baseline, with the dates it covers.
  6. The change, and the date it went live in production.
  7. Known confounders.
  8. What a null result would mean, written before the result exists.
  9. Stopping condition, and what would not count as evidence.

The last field is the one people skip and the one that does the most work. If you cannot say in advance what would not convince you, then nothing can fail to convince you.

A weak hypothesis: “adding an FAQ section will improve visibility.” Nothing disproves it. A usable one: “adding an FAQ section to page X will increase impressions for page X, measured in Search Console with page-level aggregation, over the 28 days after deployment compared with the 28 days before. If impressions do not increase, the hypothesis is wrong.” The second can be wrong, which is the only property that matters.

Our own experiment, judged by this standard

Evidence: First-party experiment Experiment #001, opened 18 September 2026 and still running, asks whether a dedicated page at /ai-seo-course provides stronger relevance for the query “AI SEO course” than a generic homepage targeting the same phrase. Held against the nine fields it is incomplete, and it is more useful to say so than to tidy it up.

Experiment #001 against the nine fields, audited 22 September 2026.
FieldState at the start
HypothesisPresent, but not in a disprovable form
Primary measurementNot designated. Six measurements listed, none primary
Observation windowNot fixed at the start
BaselinePresent as a new domain with no visibility, but no Search Console data existed until 20 September
The changePresent and dated
ConfoundersNot recorded at the start
Null result meaningNot written in advance
Stopping conditionMissing

These fields have not been retro-fitted. Where one was missing at the start it is recorded as missing. An experiment re-described after the fact to look rigorous is worse than an untidy one, because the tidying is invisible in the final write-up.

The confounder we cannot design away

The target page changed on nearly every day of the experiment, because lessons were added continuously and each addition is dated in the experiment log. A continuously edited page confounds any before and after reading of its own impressions. There is no clean before. Every week of data describes a different page from the week before it, and no analysis separates the effect of the original change from the effect of everything published since.

That is a real limitation of running a publishing experiment on the page you are publishing to. It could have been avoided by freezing the page, at the cost of having nothing to publish for a month. We chose the publishing, and the honest consequence is that Experiment #001 cannot support a causal claim about the page’s structure.

What would not count as evidence

Written before the numbers exist, and applying this project’s own observation window rule: a rise in impressions during a week when new lessons shipped, because it is confounded; any movement measured over fewer than seven days; a change in average position without the aggregation setting recorded; a single day’s ranking check from one location on one device; and anything from a tool whose limits we have not written down.

Who wrote it

Kuldeepsinh Jadeja, a developer working through modern SEO and AI search in public. Not an agency, not a certified consultant, and not selling anything. More about this site.

Page last updated .