Is your website blocking AI search? I checked 98 B2B software vendors from US buyer searches

98 B2B software vendors whose sites appeared in the organic results Frase’s SERP tool returned for 24 US-English buyer searches on 9 Oct 2026.

Is your website quietly keeping ChatGPT out? AI search tools mostly send their own crawlers to fetch your pages, and the settings that decide whether those crawlers are welcome are easy to get wrong without anyone noticing.

So I looked at how the B2B software vendors whose sites came up in Frase’s SERP tool for typical buyer searches (“best CRM software”, “zendesk alternatives”, “employee onboarding software” and so on) handle it. 98 vendors, across marketing automation, CRM, project management, HR and customer support.

Short version: none of them block AI search crawlers site-wide in robots.txt (which states what a site intends; I didn’t test firewalls), and 58 of the 93 I could check publish an llms.txt file.

There are also two traps worth checking today, including a Cloudflare change that means blocking AI training can now block Googlebot too.

First, what robots.txt actually is

Every website has (or should have) a small text file at yoursite.com/robots.txt. It’s a list of instructions for bots: which ones are welcome, and which parts of the site they should stay out of. Well-behaved crawlers check it before they do anything else, from Googlebot to the AI crawlers that OpenAI, Anthropic and Perplexity run.

Telling a bot to stay out looks like this:

User-agent: OAI-SearchBot
Disallow: /

That asks OpenAI’s search crawler not to fetch anything on the site. OpenAI says sites that opt out won’t be shown in ChatGPT search answers, though they can still appear as navigational links.

Two things to know. robots.txt is a request rather than a lock: well-behaved crawlers follow it, but actually stopping a bot is the job of a firewall or bot-protection service, and robots.txt can’t tell you what yours does. And a block can end up there without anyone meaning it to, through a plugin, a security setting or a “block AI” snippet copied from somewhere.

The AI crawlers worth knowing

Most of these companies run separate crawlers for separate jobs, and Google and Apple give you a separate robots.txt token for AI training, so you can treat search and training differently. The table below uses each company’s own description where possible.

Sources: OpenAI, Anthropic, Perplexity, Common Crawl (and here), Google, Apple (all checked 13 October 2026).

What the 98 vendors are doing

None of the 98 disallow OpenAI’s, Perplexity’s or Anthropic’s search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) site-wide in robots.txt.

(robots.txt states intent, and I didn’t test firewalls. 6 of the 98 sites didn’t serve robots.txt to my identified script, 5 of them with a bot-check page, so for those I coded a copy saved in a browser.)

Not blocking is the default, though: if robots.txt doesn’t mention AI crawlers, they’re allowed in. llms.txt is a separate file a site has to publish:

58 of the 93 vendors I could check (62%) publish an llms.txt file.

llms.txt is a plain-text or markdown file at /llms.txt meant to give AI tools a guide to a site’s most useful content (I’ve written a full explainer on it here). Some come from SEO plugins, and at least one popular plugin creates one by default, so I can’t tell you how many were a deliberate choice. Either way, it’s now common among these vendors. I didn’t measure whether it makes any difference to being cited (I’ll be looking at next). The share also depends on which vendors you look at: of the 7 I could read that Google showed for one search but Frase’s tool didn’t return, 2 had one.

The rest of the basics are close to universal. All 98 have a sitemap (listed in robots.txt or at /sitemap.xml), and 79 of the 91 homepages I could read (87%) include JSON-LD structured data in the HTML their server returns.

The few who said no, and the ones who said yes out loud

Blocking AI training is a different decision from blocking AI search, and a handful of vendors have made it. 2 of the 98 (2%) disallow at least one AI training crawler. peoplehum disallows GPTBot, ClaudeBot and CCBot. vendr disallows GPTBot, OpenAI’s training crawler, while explicitly allowing OpenAI’s search and user agents.

There’s also a newer line some sites add to robots.txt, called Content-Signal, for stating preferences about how content is used. 6 vendors use it to say ai-train=no without blocking any training crawler, so they’re stating a preference that crawlers may or may not act on. 9 use it to say yes. Zapier’s reads ai-train=yes, search=yes, ai-input=yes.

The two traps worth checking today

The first is a robots.txt rule that catches people out. A crawler that has its own section in the file ignores the general User-agent: * section completely. So a site can say this:

User-agent: *
Disallow: /

User-agent: GPTBot Allow: /

and GPTBot is still allowed in, even though the first lines look like they block everything.

The second is your firewall. If you’re on Cloudflare, it changed its defaults on 15 September 2026 (its announcement). New domains now allow AI search crawlers by default but block training and agent crawlers on pages that show ads. And on sites that block AI training, that block now also covers crawlers that do search and training together, which Cloudflare says includes Googlebot, Bingbot and Applebot, unless the owner opted out beforehand. If you’ve set Cloudflare to block AI training, through its newer AI traffic settings or the older Block AI bots option, it’s worth checking what that now covers.

What we do at Frase

At frase.io (not one of the 98 vendors in the study), our robots.txt allows every AI crawler in the table above, and names most of them explicitly. Each group also carries Content-Signal: search=yes, ai-input=yes, ai-train=no, so, like the 6 vendors above, we state a no-training preference without blocking the training crawlers. We also publish an llms.txt that maps our key pages, free tools and blog for AI agents.

What this means for you: key takeaways

1. Blocking AI search crawlers in robots.txt would make you the exception among these vendors. None of the 98 vendors in these buyer searches do it site-wide in robots.txt, so if your site does, it’s worth knowing whether anyone meant it to.

2. Opting out of AI training doesn’t have to mean opting out of AI search. They’re separate crawlers, and you can treat them differently: vendr disallows OpenAI’s training crawler while explicitly allowing its search crawler. If your team wants to keep content out of training, block the training crawlers in robots.txt rather than everything. Be careful doing it through Cloudflare, though, because its training block now catches Googlebot too (more on that above).

3. Your firewall or bot protection can turn crawlers away whatever robots.txt says. 5 of these 98 sites turned away my own identified script with a bot-check page (which doesn’t tell you what they do with AI crawlers), so ask whoever runs yours what it does with them.

4. llms.txt is now common among these vendors: 58 of the 93 I could check publish one. I can’t tell you from this study alone that it gets you cited, but if you don’t have one, you’re in the minority here, and it’s a small file to add.

Where does your site stand?

1. Open yoursite.com/robots.txt and look for the AI crawlers: OAI-SearchBot, PerplexityBot and Claude-SearchBot for search, and GPTBot, ClaudeBot and CCBot for training. A bot is blocked from the whole site if its group says Disallow: / with nothing after the slash. If the bot isn’t named anywhere, it follows the User-agent: * group instead, so User-agent: * with Disallow: / blocks it too. A bot that has its own group ignores the * group completely. A path like Disallow: /admin/ only blocks that section, and an Allow: line for a path overrides a broader Disallow: / for that path.

2. Ask whoever manages your firewall or CDN what it’s set to do with AI crawlers, especially after the Cloudflare change.

3. Check you have a sitemap, at /sitemap.xml or listed in robots.txt.

4. View your homepage source and search for ld+json. If nothing comes up, there’s no JSON-LD in the server HTML. If your site adds structured data with JavaScript, it might be there but not in the source, so check with a structured-data testing tool too.

5. Try yoursite.com/llms.txt. If you get a 404 or your homepage, you don’t have one.

You can also run the checks above in one go with Frase’s free Agent Readiness Checker, which looks at your robots.txt stance on AI crawlers, whether a firewall or bot challenge blocks the request, whether your content is in the HTML before any JavaScript runs, llms.txt, your sitemap and your structured data.

Whether ChatGPT or Perplexity name you in an answer most likely comes down to things this study didn’t measure, like what your pages say and which other sites cite you, so that’s where I’d focus.

How I ran this

Disclosure: Frase makes AI-visibility software, including the Agent Readiness Checker linked above, and the list of vendors came from Frase's own SERP tool.

Who's in it: 98 B2B software vendors whose sites appeared in the organic results Frase's SERP tool returned for 24 US-English buyer searches on 9 Oct 2026 (7-10 results per search; one of 25 planned searches was dropped for off-topic results). I spot-checked two searches against Google US. One matched closely (8 of 9 domains), and the other shared only one domain. So this is the vendors in Frase's results that day, not a copy of Google's.

How I checked: a script fetched each site on 9 Oct 2026 (12:27-12:29 UTC, from Spain, as FraseResearch/1.0, no JavaScript). I re-checked robots.txt on 10 sites a day later, and there were no changes on the 6 I could re-fetch. Because no JavaScript ran, structured data added by scripts counts as "no" (freshbooks is one example), and 8 homepages came back in non-US versions. Five sitemap results come from the browser-saved robots.txt copies.

This study doesn't measure whether these vendors are cited by AI tools. For the full methodology, get in touch.