- Agent-readiness scanners grade you against roughly twenty emerging standards. Most of them are written for software with an API, not for a website.
- Four fixes are worth making on almost any site: content signals in robots.txt, an llms.txt, markdown for agents, and a discovery catalog. Call it an hour of work.
- The rest (MCP server cards, OAuth discovery, WebMCP, agent skills indexes, the payment protocols) only mean something if an agent can call something you run. Skipping them is the right call.
- The scanner never checked our llms.txt. The file that most affects how ChatGPT and Claude describe us isn’t on the scorecard at all.
I run a school that teaches people to build with AI. Last week I ran our own website through Cloudflare’s agent-readiness scanner.
Twenty out of a hundred. Level 1: Basic Web Presence.
My first instinct was to fix everything by Friday. That instinct was wrong, and working out why turned into the most useful thing I’ve learned about this all year.
What these scanners check
The tool runs about twenty checks across five groups: discoverability, content, bot access control, API/auth/MCP discovery, and agentic commerce.
We passed three: robots.txt, sitemap.xml, and rules for the major AI crawlers. We failed everything else. No markdown negotiation. No Link headers. No API catalog. No OAuth discovery metadata. No MCP server card. No agent skills index. No WebMCP. No payment protocol.
That reads like a report card. Then I went through what the failed checks were each asking for.
Most of the rubric isn’t written for your website
Eight of the twenty checks assume you have an API for agents to call. An MCP server card describes a server. OAuth discovery metadata describes how to get a token for your protected endpoints. WebMCP exposes your site’s actions, like add-to-cart or book-the-slot, as tools an agent can invoke. An agent skills index publishes skills you’ve authored.
We have a marketing site and a Stripe checkout. There’s no server for an agent to connect to, no protected API to get a token for, no cart to add to. An MCP server card for a server that doesn’t exist is a lie in JSON.
Four more checks cover agentic commerce: x402, the Machine Payment Protocol, Universal Commerce Protocol, ACP. The scanner was straight with us there. It found no e-commerce signals and marked those informational, excluded from the score.
So of twenty checks, eight are aspirational, four don’t apply, and one is a check most sites misread. Web Bot Auth is how a company that operates a crawler proves its bot is really its bot. It’s published by the people sending requests, not the people receiving them. We don’t run a crawler, so there’s nothing for us to publish.
That leaves seven checks that genuinely apply to a normal website. That’s the real denominator.
The four things worth doing on any site
1. Content signals in robots.txt
Two years ago robots.txt could say one thing: crawl this, don’t crawl that. Content Signals splits that into three questions, because they’re genuinely different.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /search=yes means index us and link to us. ai-input=yes means use our pages to answer someone’s question and cite us. ai-train=no means don’t use our content to train model weights.
Those are three different answers for us. Being cited in an AI answer is how people find MakerSquare now. Having our copy absorbed into a training set does nothing for us. Before Content Signals there was no way to say one and not the other.
One thing to watch: robots.txt groups don’t inherit. If a bot matches a named User-agent: GPTBot group, it ignores the User-agent: * group completely. Put the signal only in * and every crawler you named by hand, the ones you care most about, never sees it. Repeat it in every group.
2. llms.txt, the one the scanner never checked
We’ve had an llms.txt for months. It’s a plain markdown summary at the root of the domain: what we are, who it’s for, what students build, prices, dates, how to enroll. When Claude or ChatGPT gets asked about MakerSquare, that file does more work than anything else on the site.
The scanner never looked for it. Not once, in twenty checks.
A score is a proxy. A useful one, but still a proxy, and this is the clearest evidence of it I’ve seen.
We also added llms-full.txt, the complete text of every public page in one document, for agents that want everything rather than the summary. Ours is 68 pages. I generate it with a script rather than maintaining it by hand, because a stale llms-full.txt is worse than none.
3. Markdown for agents
This is the one with real value beyond the scorecard.
When an agent fetches your page, it gets HTML: nav, footer, cookie banner, inline styles, three analytics scripts, and somewhere in the middle, your content. It burns tokens on markup and sometimes gets the content wrong.
Content negotiation fixes that. A normal browser sends Accept: text/html and gets your page exactly as it always was. An agent sends Accept: text/markdown and gets clean markdown of the same page. One URL, one source of truth, two representations.
If you’re on Cloudflare this is a toggle. We’re on Vercel, so it’s a small function that fetches the page and converts it.
Two things I got wrong, in case you’re doing this yourself.
The first cost me a deploy. On Vercel, rewrites are evaluated after the filesystem, which is why they work as SPA fallbacks. So a rewrite pointed at your markdown handler never fires for a page that exists as a static file. It has to be a redirect, which is checked first. Then you need a guard so the agent’s followed request doesn’t re-enter the same rule and loop.
The second one is funnier. My converter stripped <button> elements as chrome. Our FAQ page is an accordion, and every question lives inside a button. I generated a beautiful markdown FAQ containing thirty-two answers and no questions. Whatever you build, read the output before you ship it.
4. A discovery catalog
Last one, and it’s ten minutes. A JSON file at /.well-known/ai-catalog.json listing what you offer an agent. In our case that’s the llms.txt, the llms-full.txt, the sitemap, and a note that any page can be fetched as markdown. Each entry gets a few example questions it can answer, so registries can index it semantically.
Then advertise it, in the HTTP Link header on every page and with an Agentmap: line in robots.txt. A file nothing points at is a file nobody finds.
How to sort the next standard that shows up
More of these are coming. The question I use:
Does this describe content I already publish, or a service I’d have to run?
Content signals, llms.txt, markdown negotiation, a discovery catalog. Those describe things that already exist. You’re labelling reality. Cheap, honest, no maintenance.
MCP server cards, OAuth metadata, WebMCP, skills indexes, payment protocols. Those describe a service. Implementing them means building and running that service, forever. That’s a product decision with a roadmap and an on-call rotation attached. Treating it as a checkbox is how you end up with a manifest pointing at an endpoint that 404s.
If that doesn’t settle it, ask who the standard is for: people receiving requests, or people sending them. That’s the one that catches Web Bot Auth.
Where we landed
We made the four changes. The score went from 20 to 47, and the label went from Level 1, Basic Web Presence, to Level 4, Agent-Integrated. It won’t reach 100, because reaching 100 means running an MCP server we have no reason to run. Seven of fifteen scored checks, and the eight we’re missing are the eight that need an API behind them.
What we got out of it: agents can read our pages cleanly, our content preferences are stated in machine-readable terms instead of assumed, and one file tells an agent everything we offer it.
If you’ve got an hour and no llms.txt, start there. It won’t move your score by a single point.
ai-train=no with search=yes, ai-input=yes says don’t train on our content, but do index us and do cite us in answers. Those used to be a single yes-or-no. They aren’t anymore.MakerSquare is a 2-week in-person AI builder program in Austin, TX for operators, founders, and professionals - the people who understand their business well enough to build with AI, not just use it. See what two weeks of hands-on building looks like.