A client called me a few months back, half proud and half confused. Her venue page — the one page on her site that actually converts, the one with the real photos and the real pricing detail — was pulling steady organic traffic. Good rankings, decent time on page, the whole thing. But when she typed her own question into ChatGPT — "best wedding venues near Coral Gables with garden ceremony space" — she wasn't in the answer. A competitor with a thinner page and worse photos was.
She assumed the AI just liked the other studio better. It didn't. I pulled up her page the way an AI crawler actually sees it, not the way a browser renders it, and the pricing table, the venue details, half the page — gone. Blank. The page that ranked in Google and converted humans was functionally empty to the systems writing AI answers.
That gap — a page performing for people while being unreadable to machines — is more common than most business owners think, and it's almost never obvious from looking at the page in a browser. You have to look at it the way the crawler does.
Your best page can be invisible to AI crawlers for reasons that have nothing to do with quality: a robots.txt rule blocking the bot by name, a stale noindex tag, a canonical pointing somewhere else, or content that only exists after JavaScript runs — which most AI crawlers don't execute. You can check all four in about twenty minutes with curl, no tools required.
Why ranking well doesn't mean the AI can read it
Search ranking and AI citation are answering two different questions. Googlebot has crawled the modern web for over two decades and, for a large share of pages, will render JavaScript before indexing, per Google's own JavaScript SEO documentation. The crawlers behind AI answer engines are a different, younger generation of software, and most of them are far more literal. OpenAI documents GPTBot's behavior directly, and Perplexity publishes its own crawler documentation for PerplexityBot and PerplexityUser. Neither claims full JavaScript rendering the way Googlebot does. If your price, your answer, or your key fact only appears after a script runs, a real visitor sees it fine — and an AI crawler may see an empty shell.
This is the core reason a page can rank on page one and still never get quoted in an AI answer. Two completely separate systems are reading two completely different versions of the same URL.
The four ways a great page goes dark
1. Robots.txt blocks the bot by name
Every major AI crawler now publishes its own user-agent token, and each is honored differently. OpenAI's GPTBot, Anthropic's ClaudeBot, and Perplexity's PerplexityBot all state they respect robots.txt Disallow rules. A common cause of invisibility: someone (an old developer, a security-conscious host, a WordPress plugin) added a blanket bot block years ago that nobody has revisited since AI crawlers existed.
2. A stale noindex tag
Pages built on staging, then pushed live without stripping the <meta name="robots" content="noindex"> tag, are still common. Google's own documentation on the noindex directive is explicit that it blocks the page from any index built on that crawl — and any AI system whose citations draw on search index data inherits that exclusion.
3. Canonical tag pointing at the wrong URL
If your best page's canonical tag points at a different URL — a leftover from a redesign, an A/B test that never got cleaned up, or a copy-pasted template — you're telling every crawler, including AI ones, that the "real" version lives elsewhere. Traffic still flows because search engines are forgiving about ranking the URL people actually click. Citation systems are less forgiving; they tend to trust the canonical signal literally.
4. Content that only exists after JavaScript runs
This is the one that surprised my client. Her pricing table was rendered client-side by a JavaScript widget. A person's browser ran the script and displayed the table instantly. A crawler that fetches raw HTML without executing scripts got the container div and nothing inside it.
The 20-minute audit — run it right now
Open a terminal. You don't need a paid tool for this pass, just curl and your own eyes.
- Check robots.txt by hand. Visit
yoursite.com/robots.txtand search forGPTBot,PerplexityBot,ClaudeBot, andGoogle-Extended. If any has aDisallow: /or blocks the specific page's path, that's your answer. - Fetch the page as GPTBot would. Run
curl -A "GPTBot" https://yoursite.com/your-pageand read the raw output. Search it for your headline or price. If it's not there in the raw text, it isn't there for the crawler. - Repeat with PerplexityBot's user-agent. Per Perplexity's crawler docs, use their published token. Compare the result to step 2 — a page can be open to one engine and blocked from another.
- View source, not inspect element. In your browser, right-click and choose "View Page Source" (not "Inspect"). This shows the raw HTML before JavaScript runs — the same thing most AI crawlers see. If your key content is missing here but visible on the rendered page, you have a JavaScript-dependency problem.
- Check the canonical tag. In that same page source, find
<link rel="canonical">and confirm it points at the exact URL you're testing, not a redesign leftover or a tracking-parameter variant. - Check for a stray noindex. Search the source for
noindex. If it's there and you didn't put it there on purpose today, that's likely your culprit.
Twenty minutes, six checks, and you'll know exactly which of the four causes is hiding your page — instead of guessing.
What to fix, in order
Fix robots.txt and noindex first — they're binary and take minutes. Fix the canonical tag next; also fast, also binary. Save the JavaScript-dependency fix for last, since it usually means either server-side rendering the critical content or duplicating the key facts (price, answer, spec) into static HTML that loads without a script running. You don't need to rebuild the whole page — you need the two or three sentences an AI engine would quote to exist in the raw HTML.
If you're not sure whether your fixes worked, run the same curl commands again after deploying. Confirmation should take less time than the original diagnosis.
Why this hits your best page hardest
Ironically, your top-performing pages are often the most likely to carry this problem, not the least. They're the pages that got the fanciest build — the interactive pricing calculator, the JavaScript-rendered gallery, the dynamically loaded testimonial carousel. Every one of those upgrades that made the page convert better for humans is a candidate for making it invisible to machines. The plain, boring, mostly-static "About" page down the list often has zero problems, because nobody built anything fancy into it.
Questions people ask
Traffic and citation are different signals. A page can rank and get clicks from real people while still being blocked from AI crawlers by a robots.txt rule, a stale noindex tag left over from staging, a canonical pointing at the wrong URL, or content that only renders after JavaScript runs. AI crawlers like GPTBot and PerplexityBot generally do not execute JavaScript the way Googlebot does, so a page that looks fine in a browser can be an empty shell to them.
Fetch the URL with curl using each bot's real user-agent string and no JavaScript execution, then read the raw HTML that comes back. If your headline, price, or answer text is not in that raw response, the crawler cannot see it either, no matter how it renders in a browser. You can also check your robots.txt for explicit Disallow rules against GPTBot, PerplexityBot, ClaudeBot, and Google-Extended, since each engine now publishes and honors its own user-agent token.
For crawlers that state they respect robots.txt, such as OpenAI's GPTBot and Anthropic's ClaudeBot, a Disallow rule does stop that specific crawler from fetching the page. It will not retroactively remove content already indexed elsewhere, and it does not guarantee every AI system respects the same rule, since some answer engines rely on search index data licensed from a third party rather than crawling your site directly.
Want this done for you?
Want to know if AI is recommending you or your competitor? Get an AI visibility audit from Acromatico.
Get a free AI Visibility Audit →