You Can Rank #1 on Google and Be a Blank Page to ChatGPT: Why Businesses Go Invisible in AI Search

There is a specific kind of bad meeting happening in marketing departments right now.

Rankings look fine. Google Search Console looks fine. Then someone opens ChatGPT, asks the question your business exists to answer, and watches it recommend three competitors while never mentioning you. Somebody says "we need to do GEO." Somebody else suggests adding an llms.txt file. A budget gets approved for something nobody can define.

Almost none of that addresses the actual problem. The reasons businesses disappear from AI search are mostly boring, mostly technical, and mostly measurable. Better still, the biggest one is completely invisible in your Google rankings, which is exactly why it goes unfixed for years.

Here is what the evidence actually shows, and what is worth your time.

The thing almost nobody checks first

Googlebot renders JavaScript. The AI crawlers do not.

That single sentence explains more disappearances than every content strategy debate combined.

In a joint analysis with MERJ, Vercel measured crawler behavior across its network and published the results in December 2024. The finding was unambiguous: none of the major AI crawlers execute JavaScript. That list includes OpenAI's GPTBot, OAI-SearchBot, and ChatGPT-User, Anthropic's ClaudeBot, Meta's Meta-ExternalAgent, ByteDance's Bytespider, and PerplexityBot. Common Crawl, widely used as LLM training data, does not render pages either.

They do fetch JavaScript files. GPTBot pulled JS in 11.50% of requests and ClaudeBot in 23.84%. They just never run them. As Vercel put it, they "can't read client-side rendered content."

The scale involved is not trivial. In the month Vercel measured, GPTBot made 569 million fetches across its network and ClaudeBot made 370 million. Combined with AppleBot and PerplexityBot, AI crawlers accounted for nearly 1.3 billion fetches, a little over 28% of Googlebot's 4.5 billion.

Two useful exceptions. Google's Gemini runs on Googlebot's infrastructure, so it renders JavaScript fully. AppleBot also renders through a browser-based crawler. So the picture is not uniform, which is precisely what makes it confusing to diagnose.

Now put it together. If your site is a React or Vue single-page app that injects its main content client-side, Googlebot eventually renders and indexes it. You rank. Meanwhile GPTBot and ClaudeBot issue one request, receive a nearly empty HTML shell, and move on. You are simultaneously visible in Google and invisible to ChatGPT and Claude, and no ranking report on earth will tell you.

How to check this in two minutes: view the raw HTML your server returns, not the rendered DOM. In Chrome, use View Source rather than Inspect, or run curl -A "GPTBot" https://yoursite.com/your-key-page and read what comes back. If your headline, body copy, and internal links are not in that raw response, the AI crawlers are not seeing them. Server-side rendering, static generation, or incremental static regeneration all fix it. Client-side rendering is fine for view counters and chat widgets, not for the content you want cited.

There is a second finding in that data worth knowing. ChatGPT's crawler spent 34.82% of its fetches on 404 pages and ClaudeBot 34.16%, compared with 8.22% for Googlebot. AI crawlers are notably worse at URL selection. Stale sitemaps, broken redirect chains, and inconsistent URL patterns cost you far more with them than with Google.

One caveat worth stating plainly, since most articles citing this study do not. The Vercel and MERJ analysis was published in December 2024 and no independent study has replicated it at comparable scale since. Crawler behavior can change without announcement. That is an argument for testing your own site rather than trusting anyone's published figures, including these. The curl command above takes thirty seconds and tells you the truth about your site today, which is worth more than a citation.

Blocking the training bot is not the same as blocking the search bot

This is the mistake that costs the most and gets noticed the least, and you cannot understand it by reading Google's documentation, because it has nothing to do with Google.

Every major AI company runs multiple crawlers with different jobs, and they are controlled separately. Blocking one does not block the others, and blocking the wrong one removes you from the answer engines entirely. Straight from each company's own documentation:

Read that table twice, because two very common decisions sit inside it.

If you blocked GPTBot to keep your content out of training, you did not remove yourself from ChatGPT search. That is OAI-SearchBot, a separate directive. OpenAI states the settings are independent and that a webmaster can allow one while disallowing the other. Plenty of businesses think they have opted out of ChatGPT entirely and have not.

If you blocked "AI bots" wholesale at the CDN, you probably did remove yourself from all of it. ChatGPT search, Claude search, and Perplexity, in one move, usually made by whoever configured the firewall rather than whoever owns marketing.

That second scenario is now the default in some setups. In July 2025 Cloudflare announced it would block AI crawlers by default, so any new site signing up is automatically configured to deny them. Cloudflare sits in front of roughly 20% of the internet. Alongside the block it launched Pay Per Crawl, letting publishers charge for access, with Ziff Davis, The Atlantic, BuzzFeed, TIME, Quora, Stack Overflow, and O'Reilly Media among the early signatories. For a publisher with a licensing strategy that is a legitimate business decision. For a plumbing company or a regional law firm that wants to be recommended by AI assistants, it is an accident.

Two more details from the primary docs that trip people up:

Robots.txt is not the whole story, and IP blocking can backfire. OpenAI and Perplexity both recommend allowing their crawlers in robots.txt and permitting their published IP ranges, and Perplexity publishes step-by-step Cloudflare and AWS WAF configuration guidance because firewall blocks are common enough to warrant it. Anthropic warns the opposite way: blocking its IP addresses "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." If you want out, use robots.txt. If you want in, robots.txt alone is not enough.

User-triggered fetches often ignore robots.txt entirely. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules," because a human asked for that page. Your robots.txt governs automated crawling, not what happens when someone pastes your URL into a chatbot.

So check four layers, not one: robots.txt on every subdomain, CDN or WAF bot rules, host-level bot management, and whether you are allowing the search crawlers specifically rather than blocking the category. Each engine publishes its IP ranges as JSON, so verification is straightforward.

The llms.txt argument, settled honestly

This is where a lot of money gets wasted in both directions, and where most articles quote exactly one source and stop.

Google's Search Central documentation on AI features states: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."

And more pointedly: "You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add."

That gets quoted constantly as though it settles the question for the entire industry. It does not, for three reasons.

Google is describing Google. That documentation governs eligibility in AI Overviews and AI Mode. It carries no authority over ChatGPT, Claude, or Perplexity, each of which operates its own retrieval stack. "Google says you do not need it" and "no AI system uses it" are different claims, and only the first is supported.

Google is not a neutral referee here. It is competing with the same assistants you are trying to appear in, and it is the only major crawler that renders JavaScript, an advantage it holds precisely because rivals have not replicated it. Guidance that discourages the web from adopting portable, machine-readable formats is not obviously against Google's interest. That does not make the guidance wrong. It does mean it should not be treated as the last word on what every other engine wants.

And the other engines are using the format themselves. This is checkable in about a minute, so we checked rather than speculating. OpenAI's crawler documentation page opens with the line "For the complete documentation index, see llms.txt," pointing to developers.openai.com/llms.txt. Perplexity's crawler documentation page opens with "Fetch the complete documentation index at docs.perplexity.ai/llms.txt. Use this file to discover all available pages before exploring further." Both also serve markdown versions of their docs by appending .md to any page URL.

That is OpenAI and Perplexity publishing llms.txt files and instructing machines to read them, on the very pages where they document their crawlers. It does not prove their production retrieval systems consume your llms.txt, and we are not going to claim it does. It does demolish the idea that this is a fringe format the industry ignored because Google shrugged at it.

None of this makes structured data pointless either. Google still recommends it, with one instruction that gets skipped: make sure your structured data matches the visible text on the page. Mismatched markup is a liability, not an optimization.

Content-level optimization is also not folklore. The foundational academic work is "GEO: Generative Engine Optimization" by Aggarwal, Murahari, Rajpurohit, Kalyan, Narasimhan, and Deshpande, from Princeton, Georgia Tech, and the Allen Institute for AI, accepted to KDD 2024. Testing against a benchmark of diverse queries, the researchers found optimization methods could boost visibility in generative engine responses by up to 40%, with effectiveness varying significantly across domains. How you structure and source content demonstrably moves the needle. What remains unproven is that any specific file format is a prerequisite for inclusion.

That gives you a sane priority order:

1. Mandatory. The search crawlers are allowed, and they can read your content in raw HTML. Nothing else matters if this fails.
2. High value, evidence-backed. Content structured so a model can extract and quote it accurately, with clear sourcing and direct answers.
3. Cheap, plausible, unproven at scale. Machine-readable files such as llms.txt and ai.txt, plus markdown mirrors of key pages. Not required by Google. Actively used as a format by OpenAI and Perplexity in their own documentation. Low cost.

Tier three is a reasonable bet, as long as you understand it is a bet and you have already done tiers one and two. The failure mode we see constantly is a business that deployed llms.txt while its main content still renders client-side, which is like printing business cards for a shop with the door welded shut.

If you want to make that bet properly, we built the AI Readiness Kit, a free, open-source generator that produces 18 machine-readable files across seven categories, 14 of which deploy at your web root, with a plain-English guide explaining what each one does and why. It is also available as a hosted MCP server if you want your coding agent to generate them directly. We deployed the full kit for Rejuvenate Skin Spa, a medical spa in Queen Creek, Arizona, and Perplexity now cites them as a source for Plexr skin tightening queries in that market, a citation you can verify yourself rather than take our word for.

We are deliberately not claiming those files caused that citation in isolation. The same engagement fixed crawlability, structure, and markup. That is the point: the files are the last layer, not the first.

The honest framing is that GEO is not a separate discipline bolted onto SEO. It is SEO fundamentals executed for machines that are less forgiving than Googlebot, plus a modest, cheap, experimental layer on top. That is less exciting than a new acronym, and considerably more likely to work.

Writing for a click that increasingly does not come

Pew Research Center ran the study most people cite and few people read properly. Researchers tracked the browsing behavior of 900 U.S. adults across 68,879 Google searches in March 2025, with 12,593 of those producing an AI summary. Findings, published July 22, 2025, and worth labeling clearly as Google-only, since Pew noted it could not reliably identify AI summaries on other engines:

• Users who saw an AI summary clicked a traditional search result in 8% of visits. Users who did not see one clicked in 15% of visits, nearly twice as often.
• Clicks on links inside the AI summary itself happened in just 1% of visits.
• Users ended their browsing session entirely on 26% of pages with an AI summary, versus 16% without.
• Around two-thirds of all searches, with or without a summary, ended with the user searching again or leaving without clicking anything.

The strategic error is reading this as "AI is stealing our traffic" and stopping there. The more useful read is that the citation itself has become the unit of value. If your brand is named and characterized accurately inside an answer that 92% of users never click through from, that impression still happened. You are being introduced to a buyer. You just do not get a session out of it.

Google, for its part, says clicks that do arrive from pages with AI Overviews are "higher quality," with users more likely to spend more time on site. Take that with the appropriate amount of salt given who is saying it, but it points in the same direction: fewer sessions, better-qualified ones.

Cloudflare has since put a number on the same imbalance from the infrastructure side, and it is the most current measurement available because it updates continuously. Its Radar platform now publishes a crawl-to-refer ratio, dividing HTML page requests from a platform's crawlers by HTML page requests carrying that platform's referrer. For the sample week of June 19 to 26, 2025, Cloudflare reported ratios ranging from Anthropic at roughly 70,900:1 down to Mistral at 0.1:1, meaning Claude made nearly 71,000 page requests for every referral it sent, while Mistral referred ten times as many visits as it crawled.

Cloudflare is refreshingly honest about the limitation: referral counts only capture web-based tools, because traffic from native apps does not send a Referer header. As they put it, the calculations "may overstate the respective ratios, but it is unclear by how much." Treat the exact figures as directional. The direction is not in dispute. Cloudflare's own summary of the trend is that legacy search crawlers "would scan your content a couple of times, or less, for each visitor sent," and that model no longer holds.

The current numbers are live on Cloudflare Radar's AI Insights page, which is a better citation than any static figure, including this one.

Two more Pew findings deserve attention because they tell you what kind of content gets pulled in. Only 8% of one or two word searches produced an AI summary, but that rose to 53% for searches of ten words or more. Sixty percent of queries starting with who, what, when, or why produced one. And 88% of AI summaries cited three or more sources, with a median summary length of 67 words.

That is a content brief hiding in a research paper. Conversational, question-shaped, multi-clause queries are where AI answers live, and those answers are short syntheses of several sources. If your pages only target two-word head terms and bury the actual answer 800 words down under a brand story, you are not writing for the surface where the question is being asked.

Google's documentation confirms the mechanism: both AI Overviews and AI Mode use a "query fan-out" technique, issuing multiple related searches across subtopics to build a response. You are not competing for one query anymore. You are competing to be the best available answer to a dozen sub-questions you never explicitly targeted.

Building your AI presence on land you do not own

In August 2026, Reddit's share of ChatGPT Search citations collapsed from an average of 3.83% between July 18 and August 7 to roughly 0.52% by August 17, according to Promptwatch data reported by Search Engine Land. That is a decline of about 86% in four days. Google's AI Overviews showed no comparable drop.

Promptwatch was careful to say the data shows when the shift happened, not why. The working theory involves a change to how ChatGPT Search handles query fan-out.

The lesson is not that Reddit and third-party platforms are worthless. Pew found Wikipedia, YouTube, and Reddit together made up 15% of sources cited in the AI summaries it examined, so those surfaces clearly matter. The lesson is that citation share on a platform you do not control can vanish overnight, with no announcement, no changelog, and no appeal. If a large fraction of your AI visibility depends on someone else's domain, you are carrying a risk that will never show up in a status dashboard.

Own the asset. Use the platforms.

Not knowing your own baseline

The last mistake is the most common and the least dramatic: having no idea what AI systems currently say about you.

Most businesses cannot answer basic questions. Which prompts surface us? What does the model claim we do, charge, and specialize in? Is any of it accurate? Which competitors appear alongside us? Which of our pages get cited, and which never do?

Without that baseline you cannot tell whether anything you change works. You are also flying blind on accuracy, which matters more than most teams realize, because a model confidently misstating your pricing or service area is a commercial problem that no ranking report surfaces.

Google now reports AI Overviews and AI Mode traffic within the standard Search Console Performance report under the Web search type, so part of this is measurable in tools you already have. The rest requires deliberately testing prompts and logging the answers over time.

What to actually do about it

Start with rendering, because it is binary and it is the one most likely to be silently costing you everything. Pull the raw HTML for your ten most commercially important pages and confirm the substance is present without JavaScript. If it is not, that is the project. Nothing else you do for AI visibility matters until a crawler can read the page.

Then audit access per crawler, not per category, across robots.txt, CDN or WAF bot rules, and host-level bot management. The decision you are actually making is search versus training, and they are separate directives. Most businesses want OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot allowed, and can then make an independent call on GPTBot, ClaudeBot, and Google-Extended based on how they feel about model training. Inheriting a blanket vendor default makes that choice for you, badly. If you want a starting point for the robots.txt and permissions layer, the AI Readiness Kit generates those files free, along with the optional machine-readable layer.

Next, restructure how your pages answer questions. Put the direct answer near the top in plain text, before the narrative. Write self-contained passages that survive being lifted out of context, because that is precisely what a 67-word synthesis does to your content. Target the long conversational questions your buyers actually ask, since those are the queries that trigger AI answers more than half the time. Keep your structured data accurate and matched to visible text, and skip the AI-specific file formats until the platforms themselves say otherwise.

Finally, establish a baseline before you change anything else. Build a list of twenty to fifty prompts a real buyer would use, run them across ChatGPT, Google AI Mode, Perplexity, and Claude, and record what comes back including which sources get cited and whether the claims about you are true. Re-run it monthly. That log is the only instrument that will tell you whether any of this is working.

This is unglamorous work. It is also the work, and most of your competitors are buying an acronym instead of doing it.

If you want a second set of eyes, Silverback runs AI visibility and GEO audits that start exactly here: what can the crawlers actually see, what are the models actually saying, and what is the shortest path between those two answers.