Home › Guides

Is AI crawling my business website? How to check

Before ChatGPT, Claude, Perplexity or Gemini can mention what's on your website, their bots have to be able to reach it. Many small business sites block them without the owner knowing. Here's how to check in about 15 minutes, no tech skills needed.

What AI crawlers are

A crawler (or "bot") is a program that visits web pages and reads them. AI companies run their own. Some collect pages to train their models. Others fetch pages so the AI (also called SI, Super Intelligence) can answer a question with up-to-date facts and links. People sometimes call the work of getting found this way GEO (generative engine optimization).

These are the names you'll see most, taken from each company's own help pages:

The key point: training bots and search bots are separate. You can say no to training and still let the search bots in.

Check 1: Look at your robots.txt file

Type your web address with /robots.txt on the end into your browser, for example yourbusiness.com/robots.txt. It's a short text file that tells bots where they may go.

• Look for any of the names above followed by Disallow: /. That blocks that bot from your whole site.
• Also look for User-agent: * followed by Disallow: /. That blocks every bot, including Google.
• If you see "Not found", that's fine. With no file, bots are allowed by default.

A file that keeps search bots welcome but turns off training could look like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /

If you're not sure, ask whoever built your website before changing anything. One wrong line can hide your site from Google too.

Check 2: Look at your hosting or CDN bot settings

Some hosting companies, website builders and security services (such as Cloudflare) can block AI bots with a single switch, sometimes turned on by default. Your robots.txt can look fine while this switch quietly turns bots away.

• Log in to your hosting, website builder or security dashboard.
• Look for settings named "AI bots", "AI crawlers", "bot protection" or "block AI scrapers".
• Decide on purpose which ones to allow. Blocking training bots is a fair choice. Blocking search bots means those AIs can't read your pages.

Check 3: See if the bots actually visit

Your hosting company's raw server logs (often called "access logs") list every visit, including the bot's name. Search them for OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and bingbot. Some CDNs also show AI bot visits in a dashboard.

Don't rely on regular website analytics for this. Most only count visitors that run JavaScript, and most bots don't. Also note that anyone can fake a bot's name, so a single odd visit doesn't prove much. A steady pattern does.

Check 4: Make sure your pages work without JavaScript

Many bots read the plain page and don't run scripts. If your services, towns and phone number only appear after scripts load, a bot may see a nearly empty page.

• Right-click your homepage and choose View page source. Search (Ctrl+F or Cmd+F) for your phone number and a main service. If you find them, bots can likely read them.
• If they're missing, ask your web designer whether the site can show that text in the page itself.

Check 5: Sitemap and schema basics

• Sitemap: a list of your pages, usually at yourbusiness.com/sitemap.xml. Many website builders make one for you. Add a line like Sitemap: https://yourbusiness.com/sitemap.xml to robots.txt, and submit it in Google Search Console and Bing Webmaster Tools.
• Schema: hidden labels in your page code that spell out your business name, address, phone, hours and services (often called "LocalBusiness" schema). Many website builders and plugins can add it. Make sure it matches your Google Business Profile exactly.

Being crawled is step one, not the finish line

Letting the bots in only means AI tools can read your site. It doesn't mean they'll recommend you. When a customer asks for a business like yours, AI tools also weigh your Google Business Profile, your reviews, directory listings and other websites that mention you. Our guide How to get your business recommended by ChatGPT covers those. To see whether AI tools name you today, you can run your own AI prompt audit.

Or have us check where you stand

Our AI Recommendation Report ($49) asks 5 leading AIs (ChatGPT, Gemini, Grok, Claude and Perplexity) 15 real customer questions about your business, twice each. You see whether you're named, which local competitors are named instead, and which websites the AIs relied on, so you know where you stand and how you compare. The Report + Fix Kit ($99) adds ready-to-paste text written for your business, plus one re-check later.

Every report comes with a 14-day money-back promise. No one can guarantee AI will recommend you, because answers change. The report gives you a clear picture of today and a list of what to work on.

Common questions