Is Your Website Ready for AI Agents?

Agentic AI browsing has gone from a research demo to a line item on your traffic logs in about eighteen months. Claude, ChatGPT, Perplexity's Comet, and a growing list of purpose-built agent frameworks are now visiting websites not to read them for a human, but to act on a human's behalf — filling in a form, checking a price, calling an API, deciding whether your site is even worth the trip. That's a different audience with different needs, and most of the web hasn't caught up.

We've spent the last few weeks building agent discovery checks into SiteVitals, and along the way we ran our own product against the emerging standards, found real gaps in our own infrastructure, fixed them, and learned a fair amount about which parts of this space are worth taking seriously today and which are still speculation. This post is the write-up: what the industry is measuring, what we check for and why, where we deliberately draw the line, and what we think is still an open question.

The reference point: Cloudflare's Agent Readiness score

In April 2026, Cloudflare launched isitagentready.com, a free scanner that scores any domain against a growing list of emerging agent standards, and published the results as a public dataset on Cloudflare Radar's AI Insights page. It's a genuinely useful reference point — not because it's the final word on what "agent-ready" means, but because it's the closest thing the industry currently has to a shared checklist, and because Cloudflare has the vantage point (edge traffic across a huge share of the web) to say something credible about actual adoption.

Their own writeup makes the scale of the gap clear: across the top 200,000 domains they scanned, a vanishingly small number implement even the simplest standards. Markdown content negotiation, agent-declared usage preferences, MCP Server Cards — all sit in the low single digits or below. This isn't a story about a mature ecosystem with laggards. It's a story about a genuinely new layer of the web that almost nobody has built yet, which is exactly why we think it's worth engaging with now rather than waiting for it to settle.

We don't think Cloudflare's checklist is gospel, though, and we'll get into where we diverge. But it's the right starting point, and it's the one we measured ourselves against first.

Where we agree, and where we started

Cloudflare's scanner groups its checks into four areas: discoverability, content, bot access control, and API/auth/MCP/skill discovery, plus an unscored commerce category. We already had a head start on two of these before this work began — our SEO and AI-readiness audit has checked structured data quality, semantic HTML, internal linking, and llms.txt presence for a while, under the umbrella of what we call "AI visibility." What we didn't have was anything covering the newer discovery layer: the machine-readable files and headers that tell an agent what a site can do, not just what it says.

That's the gap we closed. Here's what we built, and — just as importantly — what we chose not to.

Content Signals: usage preferences, not access control

We extended our existing robots.txt checker to parse Content Signals and the IETF's competing Content-Usage directive — a layer that sits alongside traditional Disallow rules but expresses something different: not can a crawler access this page, but what may it do with the content once it has. Cloudflare introduced Content Signals as a CC0-licensed, three-category vocabulary — search, ai-input, ai-train — specifically so a site can permit indexing while declining AI training use, or vice versa.

Agent discovery surfaces: the .well-known layer

The bulk of the new work covers six well-known paths and one HTTP header, matching most of what Cloudflare's own scanner checks under its "API, Auth, MCP & Skill Discovery" category:

We deliberately show maturity honestly rather than flattening everything into one "found/not found" signal. Three of these are ratified IETF standards; three are proposals or drafts that could still change shape. Telling a site owner "you're missing this" carries a different weight depending on which category it's in, and we think collapsing that distinction — treating a three-week-old draft the same as a decade-old RFC — undersells the real standards and oversells the speculative ones.

Where presence isn't enough: spec compliance

This is the part where we think a fair number of "agent readiness" checks in the wild, including some of Cloudflare's own, stop a step too early. A file existing at the right path and parsing as JSON is not the same as it actually implementing the standard. RFC 9727 requires a specific linkset structure, not just any JSON object. RFC 8414 requires an issuer field. RFC 9728 requires a resource field whose value corresponds to the resource being described.

One field survey we came across while building this found that of 74 major API providers, only four published a spec-compliant .well-known/api-catalog — the other seventy either had nothing, or returned exactly this kind of false-positive HTML or empty response. We'd accidentally joined the second group before we caught it.

Fixing our own infrastructure was the most useful part of this exercise. It's easy to build a scanner that checks whether something responds; it's more useful to build one that checks whether the response actually means what it claims to. That's the distinction we're leaning into.

Where we deliberately don't check — yet

Three things came up repeatedly during this research that we chose to leave out of SiteVitals for now, on purpose:

DNS for AI Discovery (DNS-AID). A proposal to publish agent discovery records via DNS SVCB/HTTPS resource records, signed with DNSSEC. It's a genuinely different mechanism — DNS resolution rather than HTTP — and it's currently an individual IETF draft, not even a working-group document. For our audience of agencies and site owners who mostly don't control their own DNS zones, asking them to sign a DNSSEC-protected discovery zone is a materially bigger lift than dropping a JSON file at a well-known path. We're watching it, not implementing it.

Markdown content negotiation. Serving text/markdown when an agent's Accept header requests it, or via an /index.md URL fallback (Cloudflare's own docs site uses this pattern). We're not checking for this today. There are a couple of reasons for this. Firstly, adoption: as of early 2026, only a small number of agents — largely coding-focused ones like Claude Code — actually request markdown by default. For the kind of sites SiteVitals monitors — agency client sites, local business pages, e-commerce — the audience requesting markdown negotiation today is close to zero, and the win is mostly a token-efficiency one for documentation sites, not a discoverability one for general content. As we discussed previously in the article Is HTML Making a Comeback? Why the Age of Markdown May Be Coming to an End the main advantage of Markdown is token efficiency, and that may be less of an issue in the future than it once was. That said, this is squarely in the category of things to keep an eye on, and it's on our list to revisit as agent adoption of the Accept: text/markdown convention widens.

WebMCP. A proposed browser API (navigator.modelContext) that lets a page register callable tools directly with an agent-enabled browser. This is architecturally distinct from everything else on this list — it requires actually executing the page in a browser and checking for a JavaScript API call, not a static HTTP request. It's a real and interesting standard, currently in Chrome's early preview program, but it belongs conceptually closer to a rendering pipeline than to a .well-known scanner, and we'd rather build that properly later than bolt it on badly now.

Where this leaves us

Google's own John Mueller has publicly cautioned against treating files like llms.txt as something that meaningfully changes how large language models distinguish one site from another — a reasonable, skeptical note in a space that's prone to hype. We take that seriously.

None of this is settled. Several of the standards above are weeks old, adoption is close to zero even among the companies publishing the specs, and it's entirely possible some of them don't survive contact with reality.

Our position is that this is exactly the moment to be paying attention rather than waiting for consensus. Being early on a standard that succeeds costs you very little — most of these are a JSON file and an afternoon — and being early on genuine discoverability signals for your API or MCP server has real, if modest, upside today, independent of whether the more speculative formats pan out. What we're not willing to do is treat "the file exists" as the finish line, or pretend a three-week-old draft has the same weight as an RFC that's been through IETF review.

We'll keep this checklist evolving as the standards do — Cloudflare's Radar dashboard is a good place to watch real-world adoption trends as they shift, and we'll continue benchmarking our own checks against what they and others in this space are measuring. If you think we've got the balance wrong on any of the calls above — too cautious, too eager, missing something entirely — we'd genuinely like to hear it. This is as much a research question for us as it is a product feature.