Why AI search engines cannot see most of your website

What happens when an AI crawler fetches your page?

An AI crawler issues an HTTP request and reads the bytes that come back. That is the whole interaction. It does not start a browser engine, it does not build a DOM, it does not wait for a hydration pass, and it does not run the JavaScript bundle it just downloaded.

This is the difference between an AI crawler and Googlebot, and it is the source of nearly every surprise. Googlebot has run a full rendering pipeline for years, so a React application that renders entirely on the client can still rank in Google. That experience taught a generation of engineers that client-side rendering is safe for search. For AI retrieval it is not.

The practical consequence is specific. If a product description appears after a `useEffect` fires, it is not in the response. If a specification table lives behind a tab that mounts on click, it is not in the response. If pricing is fetched from an API after paint, it is not in the response. The crawler sees an empty shell, records that the page had nothing to say, and moves on.

How do I check whether my own pages are visible?

Fetch your page the way a crawler does and read what comes back, rather than trusting what the browser shows you. A single curl with a crawler user-agent answers the question in seconds.

Pick a sentence that carries commercial weight — a capability claim, a price, a specification — and grep the raw response for it. If the sentence is missing from the HTML, that sentence does not exist as far as AI retrieval is concerned, regardless of how prominent it looks on screen.

The failure mode this catches is not usually a wholesale client-rendered site. It is one component. A specifications accordion, a pricing table behind a toggle, a case study carousel. The page looks complete in a browser and is missing exactly the passage that would have been quoted.

VenSoc runs this as an automated build gate rather than an occasional check, because the regression is silent: a developer moves one section into a client component for a legitimate reason, nothing visibly breaks, and the page quietly stops being citable. Every route on vensoc.com is fetched with a GPTBot user-agent on every build and asserted to contain specific load-bearing sentences in the raw HTML.

Command

curl -s -A "Mozilla/5.0 ... GPTBot/1.2" https://example.com/page | grep "your sentence"

Pass condition

The sentence appears in the response body

Common false negative

Testing the homepage only, where content is usually static

Where it usually fails

Tabs, accordions, carousels, and anything fetched after paint

Does blocking GPTBot remove me from ChatGPT?

No, and this is the most consequential misunderstanding in the field. GPTBot and OAI-SearchBot are different agents with different jobs, and blocking the wrong one produces the opposite of the intended result.

GPTBot collects data that may be used for model training. OAI-SearchBot performs retrieval when ChatGPT needs current information to answer a question. A site that blocks GPTBot but allows OAI-SearchBot keeps its live retrieval visibility while opting out of training. A site that blocks OAI-SearchBot has removed itself from ChatGPT answers, whatever it does with GPTBot.

The same split exists elsewhere. ClaudeBot is the training crawler; Claude-SearchBot and Claude-User handle retrieval. Google-Extended governs AI training use and is entirely separate from Googlebot, which governs whether you appear in Search at all.

For a services business with no content moat, VenSoc’s position is to allow all of them. Presence in a training corpus is how a question like "who does Odoo customisation in Pakistan" gets answered from model memory rather than requiring a live search that may never happen. A publisher whose content is the product will reasonably decide differently — but they should decide deliberately, agent by agent, rather than pasting a blocklist.

What should a page look like if I want it quoted?

Retrieval does not work on pages. It works on passages. A system searching for an answer chunks documents, embeds the chunks, and returns the ones that match — so the unit of optimisation is a section that makes sense on its own, not a page that reads well from top to bottom.

That has concrete structural implications. Put a direct forty-to-sixty word answer immediately under the heading, before the context and the caveats, because roughly 44% of AI citations come from the first 30% of a page. Write headings as the questions a buyer would actually type. Keep each section between 150 and 300 words and make it self-contained, so that a chunk lifted out of the page still means something.

Restate entities instead of using pronouns. "Odoo’s `account.move` model" survives chunking; "it" does not, because the sentence that defined "it" may be in a different chunk. Passages that get cited heavily run three to four times normal entity density, which reads slightly stilted to a human and parses unambiguously to a machine.

And attribute every number. A statistic with a named source and a link is materially more likely to be reproduced than a bare figure, because a retrieval system has something to anchor the claim to.

What about llms.txt and schema markup?

Both are worth shipping. Neither is worth believing marketing claims about, and a firm selling you either as a citation driver is overselling.

llms.txt is a proposed convention for a plain-text index of a site’s content for language models. Adoption by the model providers is minimal — it is fetched in a negligible fraction of crawler requests. VenSoc ships one because it costs an afternoon and signals technical seriousness to a human evaluator, not because it moves retrieval.

Schema.org markup is more useful but for a different reason than usually claimed. There is no measured correlation between structured data and AI citation. What schema does drive is entity disambiguation and rich results in conventional search. For a company whose name is not a dictionary word, the `sameAs` property is the highest-value markup on the page: it is what resolves an ambiguous brand string to a single unambiguous entity across knowledge graphs.

Worth knowing when someone quotes FAQ rich results at you: Google removed them from Search on 7 May 2026 and dropped reporting support the following month. FAQPage markup remains parseable by LLM ingestion pipelines and costs nothing to keep, but it now produces zero Google visibility and must never be sold as an SEO win.

What actually predicts whether an AI engine mentions you?

Brand mentions across the wider web, by a wide margin. Published correlation studies put mentions at roughly r ≈ 0.66 to 0.74 against AI visibility, and backlinks at around r ≈ 0.22 — which is close to three times the predictive power for something most SEO budgets barely track.

That reorders the work. A listicle placement on a comparison site, a technical talk, an answer on a forum where your engineers are credibly present, an open-source repository with your name on it — these move AI visibility more than another service page targeting another keyword.

It also explains why head terms are unwinnable with on-site work. A query like "best AI development company" returns a listicle SERP, and the systems answering it cite the listicles rather than any vendor’s own page. You do not win that query by writing a better page about yourself. You win it by being in the listicles that get cited.

The on-site work still matters, but its job is narrower than it is usually sold as: make sure that when a retrieval system does come looking, the page it finds is legible, citable, and says something specific enough to quote.