# SEO in 2026: get found by search engines and AI agents

> How search engines and AI assistants find, read and choose websites, and the checklist to make yours one they can reach, understand and cite.

Published 2026-11-03, updated 2026-11-03. Source: https://witnessqa.com/learn/seo-for-search-and-ai-agents/

People now find businesses in two places: the search results and the answers AI assistants write. Both start the same way, with a crawler that has to reach your pages and read them. This guide covers what that takes, how to choose what to write about, and a 30-day plan.

## Two ways people find you now, one foundation

Someone who needs what you sell either types it into a search engine or asks an AI assistant. The search engine shows a page of results, often with an AI summary at the top. The assistant writes an answer and cites a handful of sources.

These look like different channels, but they work the same way. Before anything can rank you or quote you, a crawler has to reach your pages, read them, and store what it found:

1. **Crawl.** Automated programs download pages they find, mostly by following links from pages they already know.
2. **Index.** What they read is stored in a large database, the index. A page that is not in the index cannot appear anywhere.
3. **Rank, or cite.** When someone asks, the system picks the results or sources that look most relevant and most trustworthy.

[Google describes its search in exactly these three stages](https://developers.google.com/search/docs/fundamentals/how-search-works). AI assistants that search the web either run their own crawlers or lean on a search index. Microsoft Copilot answers from Bing's index, and ChatGPT, Claude and Perplexity each run search crawlers of their own (listed in the next section). Google's AI Overviews come from Google's normal index. [Google says](https://developers.google.com/search/docs/appearance/ai-features) there are "no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary", beyond being indexed and eligible for a snippet.

So the old work is still the foundation of the new. A site that search engines cannot read is, for the most part, invisible to assistants too. The difference is that assistants are less forgiving, as the next section shows.

## Make your site readable by agents

This is the part that changed. A person can wait for a page to load, click through a cookie banner and scroll. A crawler does none of that. It requests a URL, reads what comes back, and moves on.

### Let the right crawlers in

Each AI company now runs separate crawlers for separate jobs, and robots.txt treats each one separately. The ones that decide whether you appear in answers are the search crawlers:

| Crawler | Used for | If you block it |
| --- | --- | --- |
| Googlebot | Google Search, including AI Overviews and AI Mode | You disappear from Google |
| Bingbot | Bing, and the Copilot answers built on it | You disappear from Bing and Copilot |
| OAI-SearchBot | ChatGPT search | [Not shown in ChatGPT search answers](https://developers.openai.com/api/docs/bots) |
| Claude-SearchBot | Claude's search | [Less visible in Claude's search results](https://support.claude.com/en/articles/8896518) |
| PerplexityBot | Perplexity's search results | [Not surfaced in Perplexity results](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md) |

Training crawlers are a separate decision. GPTBot (OpenAI) and ClaudeBot (Anthropic) collect content for training models, and Google-Extended controls whether Google uses your content for its Gemini models. [Google says](https://developers.google.com/search/docs/appearance/ai-features) Google-Extended does not affect Google Search. You can block training and still appear in AI answers, as long as you leave the search crawlers alone.

The most common mistake is not a deliberate choice at all. It is a `Disallow: /` copied from a staging site, or a security plugin that blocks "bots" wholesale. Open your robots.txt and read it once.

### Put the content in the HTML

Google renders pages in a recent version of Chrome, so content that JavaScript adds after loading usually reaches Google, with a delay. Most AI crawlers do not do this. In [Vercel's analysis of hundreds of millions of crawler requests](https://vercel.com/blog/the-rise-of-the-ai-crawler), none of the major AI crawlers ran JavaScript. GPTBot and ClaudeBot downloaded script files but never executed them.

If your page is an empty `<div id="root">` until a script fills it, those crawlers see an empty page. The fix is server-side rendering or pre-rendering for anything you want found: the heading, the main text, the prices, the links. Keep client-side rendering for what happens after a person interacts.

Test it yourself by viewing the page source (not the inspector), or by requesting the page with `curl`. If your H1 and your first paragraph are not in what comes back, crawlers that do not run scripts cannot see them.

### Be unambiguous

Crawlers have no patience for guessing, so remove the guesswork:

- **One address per page.** Pick https and one host name (with or without www). Redirect everything else there in one step, and put a canonical link on every page.
- **A sitemap** that lists every page you want found, by its final URL, linked from robots.txt.
- **No stray noindex.** It often survives a launch from the staging site.
- **Structured data** (JSON-LD) that says what the page is: the organization, a product with its price, an article with its author. It must match what the page shows.
- **Clear headings, with answers right under them.** Assistants quote passages. A heading that asks "How long does delivery take?" followed by a sentence that answers it is easy to quote. The same answer buried in paragraph six is not.

### Optional: llms.txt and Markdown copies

[llms.txt](https://llmstxt.org/) is a proposal from 2024: a Markdown file at the root of your site with a summary and links to the pages an agent should read, plus clean Markdown copies of those pages. It is useful for documentation and for agents that act on your site. It is not a ranking factor, and Google says you do not need "new machine-readable files" for its AI features. Treat it as a convenience for agents, not a shortcut. This site publishes one at [/llms.txt](/llms.txt), and this article has a [Markdown copy](index.md).

### Fast and stable

A page that takes eight seconds to show its content loses people before it loses rankings. Google's thresholds are a good target: the main content appears within 2.5 seconds (Largest Contentful Paint) and the layout barely moves while loading (Cumulative Layout Shift of 0.1 or less). Measure on a throttled phone, not on your office connection.

## Understand what the searcher wants

Everything after this point is about choosing what to publish. The single idea that matters most is **search intent**: what the person typing the query actually wants. Most queries fall into four groups:

- **Informational:** they want to learn. *how to train a puppy*
- **Commercial:** they are comparing before they buy. *best dog training classes*
- **Transactional:** they are ready to act. *dog training class near me sign up*
- **Navigational:** they want a specific brand or site. *[brand name] dog training*

You do not need a tool to find a query's intent. Search for it and look at what ranks. If the top results are buying guides, Google has decided people want comparisons, and a product page will struggle there. If they are videos, you need a video. The results page is the search engine telling you what satisfied people before.

Shorter queries are usually broader, more competitive and less clear. *coffee* could mean a café, a definition, a brand or the news. *best dark roast beans for espresso* has a fraction of the searches, but every result is a comparison, so you know exactly what to write. These longer, specific queries are called **long-tail**. Each brings little traffic, the competition is manageable, and together they add up. A new site should start there.

AI summaries now appear on commercial and even transactional searches, so many people get an answer without clicking. Content that only restates what the summary already says gives no reason to click. It needs something the summary cannot give: your numbers, your experience, a tool, a clear recommendation.

## Choose the queries worth winning

A keyword is any phrase people type, from one word to a full question. Judge each one on four things:

1. **Demand.** Do people search for it at all? A topic you find important, but nobody searches for, is thought leadership, not traffic.
2. **Fit.** Is it relevant to what you sell, and where is the searcher in their journey: learning, comparing or buying?
3. **Intent.** Can you give them what the results page shows they want?
4. **Difficulty.** Can your site realistically rank, given how established it is? A new site will not beat a marketplace for *running shoes*. It may well win *running shoes for flat feet in winter*.

A query that scores well on all four is rare, so do it first when you find one.

The research itself has three stages:

- **Collect ideas.**
  - The queries you already appear for, in Google Search Console. This is the most accurate source there is.
  - The suggestions as you type a query.
  - The "People also ask" and "People also search for" boxes.
  - The words customers use in forums, reviews and support tickets.
  - Your competitors' sitemaps, to see what they cover that you do not.

  An assistant is good at turning a product category into thirty questions buyers ask. Treat that list as a starting point, not as data.
- **Validate.** Check search volume and difficulty in a keyword tool, and intent on the results page itself. Drop what fails.
- **Cluster.** Group the queries that mean the same thing, or belong to the same problem, so each group becomes one page or one series.

A query with no path to what you sell is just traffic. A free contract template can be a fine target for a document-software company, if the page leads to a sign-up. Know which stage each query serves, and what the visitor should do next.

## Build money pages first, then one topic at a time

With a long list of clustered queries, the hard part is order. Work from the bottom of the funnel up:

1. **Money pages first.** These are the pages that convert: product, service, pricing and feature pages. Their queries are transactional or commercial (*proposal software for agencies*, *buy espresso beans online*). Many sites start with blog posts because they are easier to rank, and then wonder why nobody buys. There was nowhere for the traffic to go.
2. **Then one cluster, completely.** Pick one money page and build everything that leads to it:
   - comparisons for people weighing options (*best coffee subscription*);
   - guides for people still learning (*how to brew pour-over*);
   - links between all of them and back to the money page.
3. **Then the next cluster.** Finish one before starting another.

The reason is **topical authority**. Search engines judge a page partly by whether the site covers the whole subject around it. A site that explains what ransomware is, how to spot an attack and what to do afterwards is a more credible answer to *how to prevent ransomware* than a site with one post on it. Assistants pick their sources the same way. Ten posts on ten unrelated topics build none of that.

## Write content that earns the click and the citation

[Ahrefs looked at about 14 billion pages](https://ahrefs.com/blog/search-traffic-study/) and estimated that 96.55% get no traffic from Google at all. Most of those pages are not terrible. They are just the same as everything else.

Work in four passes:

1. **Research.** Open the top five results for your query and read them.
   - What does every one of them cover? You must cover that too.
   - What do they all miss? That is your opening.
   - How long and how deep are they? That is the bar.
2. **Structure.** Write the outline before the text: an H1 that matches the query, H2s for the questions the searcher has, in the order they have them.
3. **Write.** AI tools are good at scaffolding: structure, definitions, transitions. What they cannot write is what makes a page worth citing. [Google's guidance](https://developers.google.com/search/docs/fundamentals/creating-helpful-content) asks for content made for people, says trust matters most, and warns against generating lots of text without oversight. So add what only you have:
   - what happened when you actually tried it;
   - your opinion on what is overrated;
   - your own data and customer cases;
   - quotes from experts;
   - specific numbers and real examples.
4. **Refine.** Check the page against the on-page list below, publish, and then improve it with what Search Console shows.

The bar is not "as good as the top result". It is clearly better for this exact search.

## The on-page checklist

These are small, mechanical, and the easiest wins. Run through them on every page:

- **Title.**
  - A unique title per page, with the main phrase near the start.
  - [Google sets no length limit](https://developers.google.com/search/docs/appearance/title-link) but cuts titles to fit the screen, so about 60 characters is a safe length.
  - On a guide, say what the reader gets. On a product page, say what it does for them.
- **One H1,** usually the same as the title. Then H2 sections and H3 subsections, in order.
- **A short, readable URL:** `/learn/seo-for-search-and-ai-agents/`, not `/p?id=48213`.
- **Meta description.**
  - It is not a ranking factor, but it is often your snippet, and the snippet decides the click.
  - Google [may use it](https://developers.google.com/search/docs/appearance/snippet) when it describes the page better than the page text does.
  - Write it like an ad, around 150 to 160 characters.
- **Internal links.**
  - Link from the body text to related pages, and from guides to the money page.
  - This is how crawlers find pages and understand how they relate.
  - It is one of the best effort-to-value tasks in SEO, and the one most often skipped.
- **Images.**
  - Alt text that describes the image to someone who cannot see it.
  - A descriptive file name, a modern format such as WebP, and a size close to how big it is shown.

## Earn links and mentions

A link from another site is a vote of confidence. A link from a respected publication in your field counts for far more than one from a directory. Assistants weigh sources in a similar way: being cited by sites they trust makes you more likely to be cited by them.

Link building is the hardest part of SEO, and the shortcuts (buying links, link schemes) eventually get sites penalised. What works:

- **Be the source.** Publish what others want to reference: original data, a thorough guide, a definitive resource. The most linked pages in most fields are the most useful ones.
- **Answer journalists.** Services such as Source of Sources, Qwoted and Featured send queries from reporters who need expert sources. Answer the ones in your field. It is a numbers game, but it costs about fifteen minutes a day.
- **Build a small tool.** A calculator, a template, a checker. Tools attract links because they make other people's content better, and they are now quick to build.
- **Get into the comparisons.** Questions like "what is the best X for Y" are often answered from comparison and roundup pages. Make sure the ones in your field know about you, or publish an honest one yourself.

If you are just starting, do not chase links yet. Publish ten to fifteen strong pages first, then start with journalist queries.

## Measure what you can

- **Google Search Console** shows the queries you appear for, impressions, clicks and indexing problems, straight from Google.
- **Bing Webmaster Tools** does the same for Bing, and now [reports how often Copilot and Bing's AI answers cite your pages](https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview).
- **Other assistants do not share their data.** Neither Search Console nor anything else can tell you what people type into ChatGPT or Perplexity. Check by hand instead: in a fresh, signed-out session, ask the questions your customers ask ("what is the best proposal software for agencies?") and note who is recommended. Those brands are your competition in AI answers. Repeat monthly, because answers shift.

Early signs that SEO is working come before clicks. Impressions rise, new queries appear, and positions move from page five to page three.

## A 30-day plan

**Week 1: foundations**
- Verify the site in Google Search Console and Bing Webmaster Tools, and submit your sitemap.
- Run the checklist below on your key pages, or run a [free WitnessQA audit](https://app.witnessqa.com/audit) that checks most of it for you, and fix what blocks crawlers first.
- Find 10 to 15 target queries with the four tests. Check the intent of the top five on the results page.
- List your money pages. Do they exist?

**Week 2: your most important money page**
- Write or rewrite it: research, structure, write, refine.
- Apply the on-page checklist.
- Everything you publish later will link here.

**Week 3: the first cluster**
- Publish one comparison or guide that leads to that money page, and link the two.
- Look for pages already ranking on page two or three. Improving one is often faster than writing a new one.

**Week 4: promote and measure**
- Start answering journalist queries, fifteen minutes a day.
- Ask the assistants your customers' questions and note who they recommend.
- Read Search Console for early signals.
- Plan the next cluster, not the next random topic.

You will probably not see much traffic in thirty days. You will have what most sites never get to: pages crawlers can read, the right queries, one finished cluster, and a routine for the next.

## The checklist

The same steps as an agent skill: https://witnessqa.com/learn/seo-for-search-and-ai-agents/skill.md

1. **Let search and AI crawlers in.** robots.txt must not block the crawlers that decide whether you appear in answers. Googlebot and Bingbot for search, OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity.
   Verify: Fetch /robots.txt. It answers 200 (or 404, which means everything is allowed). No group for these user agents, and no `User-agent: *` group, has `Disallow: /`.
   WitnessQA checks: [seo_robots_blocks_all](https://witnessqa.com/docs/checks/#seo_robots_blocks_all), [seo_robots_blocks_search](https://witnessqa.com/docs/checks/#seo_robots_blocks_search), [seo_robots_blocks_ai_search](https://witnessqa.com/docs/checks/#seo_robots_blocks_ai_search), [seo_robots_error](https://witnessqa.com/docs/checks/#seo_robots_error), [seo_robots_missing](https://witnessqa.com/docs/checks/#seo_robots_missing)
2. **Keep pages you want found indexable.** Remove noindex from every page that should appear in search, both the robots meta tag and the X-Robots-Tag header.
   Verify: For each key page, read the response headers and the HTML head. Neither contains `noindex`.
   WitnessQA checks: [seo_noindex](https://witnessqa.com/docs/checks/#seo_noindex)
3. **Put the content in the HTML.** The main heading, the text and the links of each page are in the HTML the server sends, not added later by JavaScript. Many AI crawlers do not run JavaScript.
   Verify: Fetch the page without running scripts. The H1 and the main text are in the response body, and so are the title, description and canonical.
   WitnessQA checks: [seo_js_only_content](https://witnessqa.com/docs/checks/#seo_js_only_content), [seo_js_only_head](https://witnessqa.com/docs/checks/#seo_js_only_head), [seo_not_rendered](https://witnessqa.com/docs/checks/#seo_not_rendered)
4. **Publish one host, over HTTPS.** Every page answers on one host name over HTTPS. The other host name (www or not) and plain HTTP redirect there with one permanent redirect.
   Verify: Request http:// and the other host name for the home page. Each answers 301 or 308 straight to the main https:// URL.
   WitnessQA checks: [seo_https_redirect](https://witnessqa.com/docs/checks/#seo_https_redirect), [seo_host_redirect](https://witnessqa.com/docs/checks/#seo_host_redirect)
5. **Keep a sitemap of the pages that matter.** /sitemap.xml lists every page you want found, by its final URL, and robots.txt points to it.
   Verify: Fetch the sitemap. It is valid XML, every URL in it answers 200 without a redirect, and the pages linked from the home page are in it.
   WitnessQA checks: [seo_sitemap_missing](https://witnessqa.com/docs/checks/#seo_sitemap_missing), [seo_sitemap_invalid](https://witnessqa.com/docs/checks/#seo_sitemap_invalid), [seo_sitemap_broken](https://witnessqa.com/docs/checks/#seo_sitemap_broken), [seo_sitemap_redirect](https://witnessqa.com/docs/checks/#seo_sitemap_redirect), [seo_sitemap_missing_pages](https://witnessqa.com/docs/checks/#seo_sitemap_missing_pages)
6. **Say which URL is the real one.** Each page has a canonical link to itself, or to the page it duplicates.
   Verify: Read `<link rel="canonical">` on each page. It exists, it is absolute, and the URL it names answers 200.
   WitnessQA checks: [seo_canonical_missing](https://witnessqa.com/docs/checks/#seo_canonical_missing), [seo_canonical_elsewhere](https://witnessqa.com/docs/checks/#seo_canonical_elsewhere), [seo_canonical_broken](https://witnessqa.com/docs/checks/#seo_canonical_broken)
7. **Write a title for each page.** A unique title that says what the page is, with the main phrase people search for near the start. Short enough that it is not cut off, about 60 characters.
   Verify: Read `<title>` on each page. It exists, it is not shared with another page, and it is neither a single word nor far past 60 characters.
   WitnessQA checks: [seo_title_missing](https://witnessqa.com/docs/checks/#seo_title_missing), [seo_title_short](https://witnessqa.com/docs/checks/#seo_title_short), [seo_title_long](https://witnessqa.com/docs/checks/#seo_title_long), [seo_duplicate_title](https://witnessqa.com/docs/checks/#seo_duplicate_title)
8. **Write a description for each page.** A unique one or two sentence summary, written like an ad for the page, about 150 to 160 characters.
   Verify: Read `<meta name="description">` on each page. It exists, it is not shared with another page, and it is not far past 160 characters.
   WitnessQA checks: [seo_description_missing](https://witnessqa.com/docs/checks/#seo_description_missing), [seo_description_long](https://witnessqa.com/docs/checks/#seo_description_long), [seo_duplicate_description](https://witnessqa.com/docs/checks/#seo_duplicate_description)
9. **Structure the page with headings.** One H1 that names the topic, then H2 sections and H3 subsections in order. Where a heading asks a question, the first sentence under it answers it.
   Verify: List the headings of each page. There is exactly one H1 and no level is skipped (no H4 straight after an H2).
   WitnessQA checks: [seo_h1_missing](https://witnessqa.com/docs/checks/#seo_h1_missing), [seo_h1_multiple](https://witnessqa.com/docs/checks/#seo_h1_multiple), [seo_heading_skip](https://witnessqa.com/docs/checks/#seo_heading_skip)
10. **Describe the business and the page with structured data.** Add JSON-LD that matches what the page shows. Organization or LocalBusiness on the home page, Product, Article or FAQPage where they fit.
   Verify: Parse every `<script type="application/ld+json">`. Each one is valid JSON with a `@type`, and the home page has at least one.
   WitnessQA checks: [seo_structured_data_invalid](https://witnessqa.com/docs/checks/#seo_structured_data_invalid), [seo_structured_data_missing](https://witnessqa.com/docs/checks/#seo_structured_data_missing)
11. **Link pages to each other.** Every page is reachable from a link on another page, related pages link to each other, and the pages that sell are linked from the pages that teach. No link is broken.
   Verify: Follow the links on each page. Every same-site link answers below 400, and every key page has at least one link pointing to it.
   WitnessQA checks: [broken_link](https://witnessqa.com/docs/checks/#broken_link)
12. **Describe every image.** Images that carry meaning have alt text that says what they show. Serve them in a modern format, at the size they are shown.
   Verify: List the `<img>` elements. Each meaningful one has a non-empty alt, and none is far larger than its displayed size.
   WitnessQA checks: [seo_image_alt](https://witnessqa.com/docs/checks/#seo_image_alt), [speed_image_format](https://witnessqa.com/docs/checks/#speed_image_format), [speed_image_size](https://witnessqa.com/docs/checks/#speed_image_size)
13. **Declare the language and the mobile layout.** Set `lang` on the html element and a viewport meta tag, and make sure nothing is wider than a phone screen.
   Verify: The html element has `lang`, the head has `<meta name="viewport">`, and on a 390 px wide screen the page does not scroll sideways.
   WitnessQA checks: [seo_lang_missing](https://witnessqa.com/docs/checks/#seo_lang_missing), [seo_viewport_missing](https://witnessqa.com/docs/checks/#seo_viewport_missing), [mobile_overflow](https://witnessqa.com/docs/checks/#mobile_overflow)
14. **Set the preview for shared links.** Add Open Graph title, description and image, so a link to the page looks right in chats and social posts.
   Verify: The head has `og:title` and `og:description`, and `og:image` points to an image that answers 200.
   WitnessQA checks: [seo_og_missing](https://witnessqa.com/docs/checks/#seo_og_missing)
15. **Make the main content appear fast.** The main content of a page appears within 2.5 seconds on a phone and the layout does not jump while it loads.
   Verify: Measure a key page on a throttled phone profile. Largest Contentful Paint is at most 2.5 s and Cumulative Layout Shift at most 0.1.
   WitnessQA checks: [speed_lcp](https://witnessqa.com/docs/checks/#speed_lcp), [speed_cls](https://witnessqa.com/docs/checks/#speed_cls)
16. **Offer a Markdown map for agents (optional).** Publish /llms.txt with a short summary of the site and links to its key pages, and Markdown copies of pages agents need. It is a proposal, not a standard, and search engines do not need it.
   Verify: Fetch /llms.txt. If it exists it starts with an H1, and the links in it answer 200.
17. **Register with the search engines' own tools.** Verify the site in Google Search Console and Bing Webmaster Tools and submit the sitemap in both. They are the only first-hand data on how you appear.
   Verify: Ask the site owner. This cannot be checked from outside.