Somebody at your association read about AI companies training on web content and asked whether anything could be done about it. Your host had a switch for that, or your security vendor did, and it got turned on. It was a reasonable decision made for a reasonable reason, nobody wrote it down, and the site kept working perfectly.
That is the setup for the most expensive silent failure available to an association website right now: blocking the crawlers you meant to block, and Googlebot along with them. Nothing breaks. The site is fast, the forms submit, uptime monitoring stays green. Search traffic simply stops arriving, and it can be weeks before anyone connects the two.
The Case That Made This Visible
In late August the IRS website dropped sharply out of Google results for high-volume queries including its own tax bracket pages, with broken favicons and missing descriptions in the listings that remained. It was reported by Search Engine Roundtable on August 27 after being flagged by several search professionals, and the site appeared to recover the following day.
Treat the cause as reported rather than established. The explanation circulating is that bot blocking intended for other crawlers also caught Googlebot. Google did not publicly confirm anything, the IRS did not comment, and we could not independently verify the mechanism. What makes it worth writing about anyway is that it is not an isolated claim: the same failure has been documented separately where AI bot blocking at the network layer swept up Googlebot as well.
The pattern is more useful than the case. An institution with a large technical staff and enormous search visibility can lose its rankings for a day or two without anyone internally noticing until outsiders point it out. An association with no in-house SEO owner would not notice at all.
Why This Is Suddenly Common
Three things changed in the last eighteen months, and they compound.
Blocking AI crawlers became a one-click option. Hosts and CDNs added toggles for it because customers asked. A toggle hides the rule it writes, which means the person who enabled it often cannot say exactly what it now blocks.
Crawler identification is genuinely hard. Blocking by user agent catches anything that declares itself honestly and misses anything that does not. Blocking by behavior, request rate, pattern, fingerprint, catches more, and also catches legitimate crawlers that behave like the thing you are filtering for. Googlebot crawls quickly and repeatedly, which is exactly the profile aggressive rate limiting is tuned to stop.
Nobody owns the consequence. The decision is made in a security or hosting context. The damage appears in a marketing metric. In most associations those are different people, different vendors and different meetings.
The Check, Which Takes About Ten Minutes
This is cheap to rule out and expensive to miss, which is an unusual combination and the reason it is worth doing this week rather than adding to a list. Four steps, in order:
- Open Search Console and look at Crawl Stats. Settings, then Crawl stats. You are looking for a drop in successful requests, or a rise in the share of responses that are not 200. A cliff rather than a slope is the signature of a rule being switched on.
- Run URL Inspection on three live pages. Your homepage, one deep membership page, one blog post. Use Test Live URL, not the cached result. If Google cannot fetch the page right now, it tells you so plainly.
- Read robots.txt with your own eyes. Not the version in a plugin's settings screen. The file as served. Security tools and migration scripts both write to it, and a staging directive that shipped to production is a classic.
- Ask your host or security vendor what is currently blocked, in writing. Specifically whether any AI crawler rule, WAF ruleset, rate limit or bot management product has changed in the last twelve months, and whether Googlebot is explicitly allowed. Explicitly is the operative word; not blocked by name is not the same as allowed.
What Verification Actually Means
Googlebot can be verified. Google publishes the IP ranges its crawlers use, and a correctly configured rule checks the request against those ranges by reverse DNS rather than trusting what the visitor claims to be. That distinction is the whole game: a rule that trusts the user agent string can be fooled by anything, and a rule that ignores it entirely can block the real thing.
This is the specific question to put to a vendor. Not whether they block AI crawlers, which they will happily confirm, but how they verify legitimate crawlers before applying a rule. A vendor who cannot answer that has not thought about your search traffic, which is fair enough, because it is not what you bought them for.
The Two Places It Usually Hides
When a site is blocking Googlebot without meaning to, it is almost never in robots.txt, because that file gets looked at. It is in one of two layers nobody thinks of as content settings:
The edge. A CDN or WAF rule applied before the request reaches your server. Your CMS has no knowledge of it, so nothing in WordPress or Drupal will show you it exists. This is where AI-crawler toggles live, and it is the layer most likely to have changed without a ticket.
The rate limiter. Not a block at all, which is why it evades a yes-or-no audit. It returns 429 or 503 to a crawler that requests too quickly, and Googlebot responds by slowing down and eventually crawling less. The site stays up, the rule looks like it is working, and indexing quietly decays over weeks rather than failing on a single day.
Deciding About AI Crawlers on Purpose
Worth separating from the accident. There is a real decision underneath, and associations land in different places on it for good reasons.
Blocking has a cost that is easy to overlook. If your members increasingly find answers through AI assistants, being absent from what those systems can read is a visibility decision, not a neutral one. If your content is a membership benefit that people pay for, blocking may be exactly right. Those are different organizations and they should not have the same setting.
Either way it should be a decision with a name attached. Written down, with who asked for it, what it blocks, and when it was last reviewed. Most of the harm here comes from settings nobody remembers choosing.
When You Can Skip This
If nothing has changed on your hosting or security stack in the last year, no bot management product has been added, and your Search Console crawl stats are flat, you are fine and this is ten minutes you can keep. The check is worth running once precisely so you can stop wondering.
Have Us Run the Crawl Check
Give us Search Console access, or just the site address and permission to test it. You will get back a written page: whether Googlebot can currently fetch your key templates, what your robots.txt actually serves, whether crawl stats show a break, and what your host or CDN is blocking at the network layer. If everything is clear, that is what the page will say, and it takes one email to find out rather than a quarter of unexplained traffic decline.