Why is my robots.txt file different to the one I created?

A while back, while setting up an early version of SiteVitals, I ran into a problem that's a good reminder there's always something new to learn online.

I'd written a robots.txt file by hand, deliberately allowing most AI crawlers in. Then no matter how hard to tried, the bots claimed they weren't allowed.

I checked the file in the repo and it looked fine. I asked again, but got the same response. They were being blocked.

It wasn't until I manually checked what was actually being served by the website, that things got more confusing. My live robots.txt file WAS apparently blocking some bots.It was quite different from the file sitting on the webserver. I puzzled over this for a little while until it struck me. Cloudflare, sitting in front of the site handling DNS, had its own managed robots.txt feature switched on — and it was quietly adding its own rules ahead of mine, blocking the exact crawlers I'd gone out of my way to allow.

It's a great example of how something can be broken without you even realising. Nothing had thrown an error. My file still existed, untouched, exactly where I'd put it. It was just being ignored and overwritten.

Why this happens

Cloudflare's managed robots.txt feature is genuinely useful, and a lot of site owners turn it on, sometimes without quite realising what it does. If your site doesn't have a robots.txt file, Cloudflare creates one with rules blocking known AI training crawlers. If you already have one, it prepends its own rules ahead of yours rather than replacing it outright. Either way, the file a crawler actually receives can differ from the file sitting in your codebase, and there's no obvious signal that this has happened unless you go looking.

This isn't a flaw in Cloudflare's design. The controls behind it are detailed and well thought through: you can allow specific crawlers, block others, see which ones are respecting your directives versus ignoring them, and even charge for access in some cases through their AI Crawl Control product. The problem isn't the feature. It's that it lives in a dashboard, separate from your codebase, and it's easy to forget it's there, or not know it was ever switched on, especially if someone else set up your account or it came with sensible-sounding defaults you didn't read closely.

If Cloudflare is managing part of your file, it's not subtle once you know what to look for. It inserts a clearly marked block bounded by # BEGIN Cloudflare Managed content and a matching # END Cloudflare Managed Content comment, along with a documented Content Signals Policy block. The trouble is that almost nobody opens their robots.txt file and reads it line by line. It's not the kind of thing that ends up on a checklist.

It's not just "block or allow" anymore

The other thing that's changed is that "AI crawler" stopped being one category a while ago. There's now a real, meaningful difference between a crawler that's training a model and a crawler that's answering someone's question right now. OpenAI runs GPTBot for training and a separate OAI-SearchBot for live ChatGPT search results. Anthropic has the same split. Blocking the first doesn't block the second, and treating them as one bot can mean accidentally shutting out exactly the traffic you wanted.

Cloudflare's own taxonomy for this is the clearest framing we've seen: every AI crawler falls into ai-train (training or fine-tuning a model), ai-input (real-time use of your content to answer a specific question), or conventional search indexing. It's a good enough distinction that we've adopted it directly rather than inventing our own.

None of this means there's a universally correct answer. Blocking a training crawler is a perfectly reasonable choice if you don't want your content feeding someone else's model. Blocking a search/answer crawler is a different decision with different consequences, and it's worth knowing which one you're actually making.

What we changed

Off the back of running into this ourselves, we rebuilt how SiteVitals checks AI crawler access. A few specific things changed:

• Blocking a crawler is no longer treated as a defect. Our check used to score any blocked AI crawler as at least a warning. That was the wrong instinct. Blocking is often a deliberate, sensible choice, so the check now reports what's blocked and what's permitted without passing judgement on the decision itself.
• Crawlers are now categorised as training or search/answer crawlers, using Cloudflare's taxonomy, so the report can tell you "you're blocking training crawlers but allowing search crawlers" instead of just a flat count.
• We now check for a managed-robots.txt fingerprint. If Cloudflare's managed block is present in the file, SiteVitals flags it directly, with a note that part of the policy may not be coming from your own configuration — which is exactly what would have saved me the afternoon I lost to this originally.
• We added a check for global wildcard blocks. If a site blocks every crawler outright (User-agent: *, Disallow: /), the report now says so plainly, rather than implying a deliberate, AI-specific policy when the real story is closer to "everything is blocked, AI included."

The list of crawlers we check against was also brought in line with Cloudflare's own default set, which added a few we weren't previously tracking, including Common Crawl's CCBot and Meta's current crawler.

What I'd actually check

If you're running anything through Cloudflare, or any other DNS or CDN provider with similar bot-management features, it's worth occasionally checking the robots.txt a crawler actually sees, not just the file in your repository. They're not always the same file. A quick way to spot the difference: open your robots.txt in an incognito browser tab and look for a block of comments mentioning content signals or AI bot management that you didn't write yourself.

This is part of why we check AI crawler accessibility continuously rather than as a one-off, and why we built the changes above. The point isn't to second-guess Cloudflare's controls, which are good and worth using deliberately. It's that "deliberately" is the key word. A change made on purpose, with full knowledge of what it does, is fine. A change that happened by default, three months ago, that you've never seen, is the kind of thing worth knowing about before someone asks why your content has gone quiet in AI search results and you have no idea where to start looking.