Two AI assistants couldn’t read our website. Here’s why

Recently, two of the four major AI assistants could not read our website. Perplexity and Claude were refused when they tried to fetch tusglobal.tech, while ChatGPT and Gemini summarised it without complaint. The cause was two firewall rules on our hosting provider's servers, and the fix was four lines in a configuration file. Several client sites we host were affected in the same way.

We caught it during our routine checks of how AI assistants read the sites we look after. This is how we traced it, because the obvious checks all said everything was fine.

What we noticed

We regularly ask the major assistants to fetch and summarise the sites we manage. ChatGPT and Gemini returned accurate summaries of ours. Perplexity said it couldn't access the site. Claude's fetcher reported that the site "disallows automated access".

That matters more than it sounds. An assistant that can't read a page can't quote it, cite it or recommend the business behind it. Two of the four assistants your customers might use were effectively looking at a closed door.

It's also easy to miss. An assistant can answer from a search engine's cached copy of your site rather than fetching it live, so a site can look fine in one assistant while being unreachable to the next.

How we narrowed it down

Each check ruled one cause out.

1. The robots.txt file. This is the file that tells crawlers what they may read. Ours explicitly allows every major AI crawler, and it hadn't been edited since 10 September, the last date we knew Claude could read the site. So the file wasn't the problem.

2. The crawlers' names. We requested our own pages from our own connection while pretending to be each AI crawler, by sending their user-agent names. Every request returned a normal page. So the server wasn't simply blocking those names.

3. The CDN. Our host puts a content delivery network in front of the site. We switched it off completely. The assistants were still refused, so the CDN wasn't the cause either.

4. A control on another host. We have an app running on a separate server with a different provider. Claude's fetcher read it without any trouble, while every site on our main hosting was refused. Same fetcher, same day, one difference: where the site was hosted.

That last test was the turning point. It moved the question from "what's wrong with our site?" to "what's different about this server?"

One note on method: Perplexity's on-request fetching gave inconsistent results, failing even on the other host, so we relied on the fetcher that behaved predictably and kept Perplexity out of the evidence.

The cause

Our hosting provider's first-line support checked the account and found no blocks on it. We went back with the evidence above, including the timestamps of failed attempts. Their support then identified two rules in the server's firewall, ModSecurity, that were refusing requests from these AI services. Because the rules run on the server itself, nothing in the hosting control panel showed them, and turning off the CDN made no difference.

Why our own test requests got through while the real crawlers didn't, we can't say with certainty. Our requests came from an ordinary office connection rather than from the AI companies' own servers, which is the most likely difference. It is also why testing your site from your own computer can give you false reassurance.

The fix

The support team's fix was to switch those two rules off for our site, in the .htaccess file at the root of the website:

<IfModule mod_security.c>
SecRuleRemoveById 400011
SecRuleRemoveById 400012
</IfModule>

After the change, Claude's fetcher could read our site again, and so could a client site patched the same way.

Two cautions before you copy it:

• Those rule numbers belong to our provider's firewall. On another host they will do nothing, or refer to something else entirely. If you see the same symptoms, ask your host which rules are responsible rather than pasting ours.
• Firewall rules exist for a reason. These two also filtered some genuinely unwanted bots. Removing them was the right trade for sites that need to be read by AI assistants, but we're watching traffic and form spam on the affected sites for the next few weeks.

Why a perfect robots.txt wasn't enough

Before an AI crawler reads any page, it first reads robots.txt. In our case the firewall stopped it at that very first step, so the crawler never saw the file that said it was welcome.

The robots standard (RFC 9309) says that if a server errors when asked for robots.txt, the crawler must treat the whole site as off-limits. Some AI fetchers go further and refuse whenever robots.txt can't be read at all. Either way the result is the same: a correct robots.txt counts for nothing if the server blocks crawlers before they reach it.

How to check your own site

• Ask the assistants directly. Ask Perplexity, Claude, ChatGPT and Gemini each to fetch your homepage and your robots.txt and tell you what they contain. A refusal from any of them is worth investigating.
• Don't stop at one. An assistant may answer from a cached copy, so one success proves less than it seems.
• Don't trust a test from your own computer. Your connection isn't where the crawlers come from. Ours passed every local test while the real crawlers were refused.
• Use a control. If you have any site on a different host, test it the same way at the same time. If one works and the other doesn't, look at the hosting, not the site.
• Repeat it. A block can appear without any change on your side, as ours did. A check that runs once a year will miss it for most of the year.

What we changed

We're applying the same fix to every site we host with this provider, and checking each one the same way afterwards rather than assuming it worked. Testing whether the AI assistants can actually read each site, rather than only reading its robots.txt, is now a standing step in the routine maintenance we run for the sites we manage.

If you'd like us to check whether the major AI assistants can read your website, get in touch. Our managed IT service includes this kind of monitoring, and the search visibility work builds on it.