Skip to content

A blanket block on all AI crawlers hits search too.

A comparison using the example of a trade publisher whose articles never show up as sources in ChatGPT search or on Perplexity, even though Google finds them easily. A contractor had blocked every known AI crawler in robots.txt. The table shows, for each crawler, what it collects content for and what management decided: GPTBot, which collects content for model training, stays blocked; OAI-SearchBot for ChatGPT search and PerplexityBot for search on Perplexity are allowed. The rule is documented with a date.

A traditional search engine returns a ranked list of links. The searcher does the reading and comparing. In AI search, a language model takes over that step: the system finds relevant pages, passes the retrieved passages to the model, and shows a written answer, usually with links to the pages it used.

Well-known examples are AI Overviews and AI Mode in Google Search, search in ChatGPT, Perplexity, and Microsoft Copilot, which grew out of the 2023 chat in Bing. They differ in index, layout, and source selection. The principle is the same: the answer rests on retrieved sources instead of the model's training data alone.

For companies, this changes how they get found. A website can appear as a source in an answer without anyone visiting it, or be missing even though it ranks high in the classic results.

Chapter 1 of 7

How an AI search answer comes together.

First, the system interprets the question, for example “What does an employer have to do for a risk assessment of computer workstations?” It breaks complex questions into sub-questions and runs several searches at once, here on the legal basis, the process, and documentation. Google calls this query fan-out.

Next, the system retrieves matching pages or passages from its index. A language model writes the answer from those passages, and the interface shows it with links to the pages it relies on.

Technically, this is retrieval-augmented generation (RAG) across the web. The goal is called grounding: the answer should be tied to sources found at the moment of the query. That lets AI search answer questions about recent events the model never saw in training.

Chapter 2 of 7

The major services differ in index and presentation.

Google shows AI Overviews as a summary above the classic results. AI Mode is a separate area of Search for longer questions in a conversation. It launched in Germany, Austria, and Switzerland on October 8, 2025.

OpenAI introduced search in ChatGPT on October 31, 2024. ChatGPT decides from the question whether to search the web, or the user turns search on. Sources appear as links and in a sidebar. According to OpenAI, search draws on third-party search providers and on content from partners.

Perplexity is a standalone search service that writes answers with citations.

Microsoft introduced a new Bing in February 2023 that summarizes results from across the web, cites its sources, and offers a chat for follow-up questions. In November 2023, Microsoft renamed Bing Chat to Copilot.

Chapter 3 of 7

The answer replaces some of the clicks.

In a results list, the searcher decides which pages to open. In AI search, the system preselects and summarizes. Many questions are answered before anyone visits a website.

The Pew Research Center analyzed the Google searches of 900 U.S. adults in March 2025. Users who saw an AI summary clicked a traditional search result in 8 percent of visits, compared with 15 percent without a summary. They clicked a link inside the summary itself in 1 percent of visits.

Google says clicks from results pages with AI Overviews are higher quality, because users tend to spend more time on the sites they visit. The two claims measure different things: Pew counts clicks, Google describes their quality. Google also says the sub-searches let it show a wider and more diverse set of links than classic search.

Chapter 4 of 7

How a website qualifies as a source.

Google Search Central says there are no additional technical requirements for Google's AI features. A page has to be indexed and eligible to appear in Search with a snippet. With the nosnippet, data-nosnippet, and max-snippet controls, a site owner limits what Google shows from its pages, including in AI Overviews and AI Mode.

Other providers run their own crawlers, each controllable in robots.txt. At OpenAI, OAI-SearchBot surfaces websites in ChatGPT search, while GPTBot collects content for training. Blocking only GPTBot keeps content out of training but doesn't remove it from search. Perplexity draws a similar line between PerplexityBot for search and Perplexity-User for fetches a person triggers with a question; according to the company, those fetches generally ignore robots.txt.

A blanket block on all AI crawlers therefore hits search too. It is better to decide on training and search separately.

Chapter 5 of 7

Visibility in AI answers can only be measured in part.

Google counts appearances in AI Overviews and AI Mode in the Search Console Performance report, under the “Web” search type. They show up there together with the classic results.

The answers themselves are generated fresh for every query. Whether sources appear, and which ones, depends on the wording and on the conversation so far. ChatGPT, for example, decides from the question whether to search the web at all. A manual spot check therefore shows only a slice.

How companies can track their visibility in AI answers systematically and work on content that gets cited is covered on the page about AI visibility. Rankings in the results list now describe only part of what customers see when they search.

Chapter 6 of 7

A citation shows where a text came from, not that it is right.

A link under an answer doesn't prove the model represented the source correctly. Language models can shorten statements, blend two sources, or add something that appears in neither.

For business decisions, anyone passing on an answer from AI search should open the cited source and check the passage. For law, taxes, medicine, and prices, only the original source counts.

Confidential material doesn't belong in public AI search. For internal documents, there are enterprise search systems that search only approved content and respect access rights.

A usable answer ties every important statement to a source you can open, and the sources are current. Ideally they come from responsible bodies such as agencies, manufacturers, or trade associations.

Chapter 7 of 7

Where the term stops working

AI search describes a way of searching, not a single product; what holds for one service doesn't automatically hold for the others. It differs from semantic search, which finds by meaning but writes no answer, from a chatbot answering from training data without a web search, and from an AI agent that goes beyond answering and acts in other systems. Working on visibility in these answers is called generative engine optimization (GEO).

Frequently asked questions

What is AI search?

AI search is web search in which a language model retrieves relevant pages and writes an answer from them, with links to its sources. Instead of a results list, the person gets a summary and can ask follow-ups.

What AI search engines are there?

Common ones are AI Overviews and AI Mode in Google Search, search in ChatGPT, Perplexity, and Microsoft Copilot. They use different indexes and pick sources by their own rules.

How is AI search different from a traditional search engine?

A traditional search engine returns a ranked list of links; AI search returns a written answer with citations. The system makes the preselection the person searching would otherwise make. Checking whether the sources support the answer is still up to people.

How is AI search different from semantic search?

Semantic search finds content by meaning rather than matching words and still returns a list of results. AI search often uses such methods for retrieval and then also writes an answer from the passages it found.

Can AI search give wrong answers?

Yes. The language model can misrepresent sources or add statements that appear in none of them, and outdated pages lead to outdated answers. A link shows where a text came from, not that it was carried over correctly.

How does a website become a source in AI search?

On Google, a page has to be indexed and eligible to appear with a snippet; Google says there are no additional technical requirements. For ChatGPT and Perplexity, robots.txt should allow the providers' search crawlers, OAI-SearchBot and PerplexityBot. None of this guarantees a citation.

    ↑ ↓ select · Enter open · Esc close