How do I know if my website is actually invisible to AI crawlers?

View-source on your homepage, not "Inspect Element": the raw HTML response. If content you can see in the browser isn't in that raw source, a non-JavaScript crawler isn't seeing it either. Google's Rich Results Test and disabling JavaScript in your browser are two other quick ways to check.

Could I be blocking AI crawlers by accident?

Yes, more often than you'd think. Security plugins, CDN bot-protection defaults, and robots.txt templates written before GPTBot or ClaudeBot existed can block them without anyone deciding to. It's worth checking on purpose rather than assuming "allow all" is still your setting.

Is adding an llms.txt file worth doing?

It's cheap enough to be worth an hour, but don't expect much from it yet. Adoption is low, and Google's John Mueller has said none of the major AI services have confirmed they read it. Ship it as hygiene, not as a strategy.

If I fix all ten of these, will ChatGPT start citing my business?

There's no guarantee. Citation behavior varies a lot by platform; only about 11% of domains get cited by both ChatGPT and Perplexity, for example. What this checklist actually does is remove the blockers that make citation impossible. It doesn't promise a result; it just stops ruling you out before you had a chance.

Can AI Even See Your Website? The 10-Point Checklist I Run on Every Client Site

Field Notes is our ongoing look at what's actually happening in AI and what it means for small businesses in Spring, TX and across Greater Houston. No hype, just the signal.

I start every new client audit with the same question: can AI actually see this website? Ask ChatGPT, Claude, or Perplexity about a small business and watch what comes back. More often than not, for a business running a modern, JavaScript-heavy site, the answer is close to nothing. Not because the business doesn't have a website, but because the one they built is invisible to the bots deciding whether they get mentioned at all.

That's not a rare problem. It's the default state of most small business websites built the modern way: fast, JavaScript-heavy, great-looking in a browser, and completely unreadable to the bots that increasingly decide whether you show up in an AI answer at all.

This is the exact checklist I run on every site I audit, in order of how much it actually matters.

Why This Is Suddenly Worth Your Time

Most AI crawlers (GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, OAI-SearchBot) don't execute JavaScript. If your site renders its content client-side, the way most React, Vue, and Angular sites do by default, these bots request a page and get back an empty container with nothing in it. Your homepage can look perfect in Chrome and be completely blank to the bot deciding whether to cite you.

The payoff for getting this right is real, too. Ahrefs' analysis of 75,000 brands found that branded web mentions correlate with AI visibility roughly three times more strongly than backlinks do. Muck Rack's audit of over a million AI citations found that about 94% come from non-paid, earned sources: not ads, not directories, just content AI models found and trusted enough to quote. None of that is possible if the bot never sees your content in the first place.

The 10-Point Checklist

1. Serve real HTML, not a JavaScript shell. This is the one that blocks everything else. If your site is client-rendered, migrate to static site generation or server-side rendering so every route ships fully-formed HTML on the first response. This is usually the first fix I make on a new client audit: move the site to a static-generation build that pre-renders every page. Same dev workflow, but now every route has real content whether or not the visitor's browser runs a single line of JavaScript.

2. Ship a real sitemap.xml. Not a JavaScript app pretending to be one. An actual XML file listing every real URL, generated fresh at every build. If yoursite.com/sitemap.xml returns your homepage instead of a sitemap, that's a soft-404 that confuses every crawler that hits it.

3. Let AI crawlers into your robots.txt, on purpose. Plenty of sites accidentally block GPTBot, ClaudeBot, and PerplexityBot through an overzealous security plugin or a robots.txt template that pre-dates AI crawlers entirely. Check yours. "Allow all" isn't the default it used to be.

4. Give every page its own title, description, and canonical tag. A single sitewide title copied across every route, or every canonical tag quietly pointing back at your homepage, tells search engines your inner pages don't matter. Each page needs its own.

5. Add an og:image, sized correctly, to every page you'd want shared. 1200×630 pixels is the standard social and AI-preview size. Without it, a shared link to your site shows up blank in Slack, LinkedIn, and iMessage, which quietly kills the "branded mention" signal Ahrefs found matters so much.

6. Add structured data: LocalBusiness, Article, Person, and FAQPage. JSON-LD schema is the most direct way to hand a machine an unambiguous description of who you are, what you publish, and who wrote it. FAQPage specifically stopped producing the little accordion in Google Search results as of May 2026, but Google has confirmed it still uses the data to understand the page, and it's still read by Bing and PerplexityBot. Keep it; just don't expect a rich-result badge from it anymore.

7. Lead every article with a direct, self-contained answer. Don't make a reader (or a language model) dig through three paragraphs of setup to find out what the page is actually about. A 40–60 word "quick answer" at the top, stated plainly, is exactly the shape an AI model wants to quote back to someone.

8. Write real, visible FAQ sections, not just schema. The schema and the visible content should say the same thing, because engines cross-check both. A FAQPage block with no matching on-page text looks synthetic; real, readable Q&As that happen to also be marked up properly look legitimate, because they are.

9. Back claims with named, specific sources. "Studies show AI saves time" convinces no one, human or machine. "The Ramp AI Index found 43.5% of U.S. businesses now pay for Anthropic subscriptions or tokens" is the kind of specific, attributed claim that gets pulled into an AI-generated answer as a citation.

10. Go check what the crawlers actually see. Don't assume. View-source on a real page. Run it through Google's Rich Results Test. Load the page with JavaScript disabled. The gap between what you built and what actually shipped is exactly where most of these problems hide. It's where I find nearly every issue on a new client audit, and it takes about ten minutes to check.

What This Actually Looks Like in Practice

The pattern shows up the same way almost every time: no sitemap, no og:image anywhere, every route rendering the same client-side shell, a canonical tag quietly pointing every page back at the homepage. None of it is visible to a human clicking around the site in a browser. All of it is invisible in a different, more consequential way, to the crawlers deciding whether the business exists.

Fixing it is rarely a redesign. It's working down a list very close to this one, in this order.

Where to Start If You Only Fix Three Things

If your site is client-rendered JavaScript with no static HTML fallback, that's first. Nothing else on this list matters until a crawler can actually read a page. After that: a real sitemap, and a robots.txt that isn't accidentally blocking AI bots. Those three alone move you from invisible to readable. Everything past that is about earning the citation, not just making it possible.