Agent readiness checklist: every check, who needs it, and how to test it

How to use this checklist

The checks below follow the groups in the isitagentready.com scanner. Cloudflare's dashboard has a similar Agent Readiness scan, announced on its blog on 6 August 2026, which sorts much the same checks by effort and leaves the commerce checks out of the score. Each check says what it is, which spec defines it, who should implement it, who should not, a minimal correct implementation and a curl command to test it. Replace example.com with your domain.

Work top to bottom and stop when a check does not apply to you. A scanner counts a missing OAuth server against you, but publishing metadata for a service you do not run is worse than a lower score: agents will try to use it and fail. We left four checks failing on our own site on purpose, and a fifth (DNS-AID) waits on a DNS change; the reasons are at the end.

All 23 checks at a glance

Our own before and after scans

Before: citerlabs.com on 28 October 2026, before any of this work. After: the preview deployment of the same site with these changes (agent-ready.citerlabs-studio-2jp.pages.dev), scanned the same day. The after column is a preview, not production, until the changes are released and the scan is re-run on citerlabs.com.

Two rows differ for reasons that have nothing to do with our changes: the A2A agent card is served on citerlabs.com by a separate Cloudflare Worker that does not run on preview hostnames, and the DNS-AID check on a pages.dev hostname reads Cloudflare's own records, not ours.

Discoverability: robots.txt

What it is: a plain-text file at /robots.txt that tells crawlers which paths they may fetch. Spec: RFC 9309. A named User-agent group replaces the * group for that crawler, so repeat your Disallow lines in every named group.

Who should implement it: every site. Who should not: nobody, but do not use it to hide private pages; it is public and only a request.

Evidence: OpenAI, Anthropic and Perplexity document that their search crawlers obey robots.txt, so blocking them keeps your pages out of those answers. That is the one link to AI visibility in this list that vendors state themselves.

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

# Test (expect 200 and text/plain)
curl -sI https://example.com/robots.txt

Discoverability: sitemap.xml

What it is: an XML list of your canonical URLs, referenced from robots.txt. Spec: sitemaps.org protocol.

Who should implement it: every site with more than a handful of pages. Who should not: nobody. Evidence for AI visibility specifically: none found; it helps any crawler find pages.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://example.com/</loc><lastmod>2026-10-28</lastmod></url>
</urlset>

<!-- Test -->
<!-- curl -s https://example.com/sitemap.xml | head -5 -->

Discoverability: Link response headers

What it is: HTTP Link headers (RFC 8288) on your pages that point agents at machine-readable resources, using registered relations: api-catalog (RFC 9727), service-desc, service-doc, describedby and alternate.

Who should implement it: sites that actually have the files the headers point to. Who should not: do not point service-desc at an OpenAPI file you do not have. Evidence: none found for AI visibility.

Link: </.well-known/api-catalog>; rel="api-catalog", </openapi.json>; rel="service-desc"; type="application/vnd.oai.openapi+json", </docs>; rel="service-doc", </llms.txt>; rel="describedby", </index.md>; rel="alternate"; type="text/markdown"

# Test
curl -sI https://example.com/ | grep -i '^link'

Discoverability: DNS for AI Discovery (DNS-AID)

What it is: SVCB or HTTPS DNS records under _agents (for example _a2a._agents.example.com or _mcp._agents.example.com) that advertise agent endpoints, signed with DNSSEC. It is an Internet-Draft, not a standard.

Who should implement it: sites that run an A2A agent or MCP server and can turn on DNSSEC. Who should not: sites with no agent endpoint. Evidence: none found. We have not done this yet on citerlabs.com: it needs DNSSEC on the zone, which the domain owner has to enable.

_mcp._agents.example.com. 3600 IN SVCB 1 example.com. alpn="h2" port=443

# Test
dig +dnssec _mcp._agents.example.com SVCB

Content accessibility: Markdown for agents

What it is: when a request sends Accept: text/markdown, the server returns a Markdown version of the page with Content-Type: text/markdown, Vary: Accept and, optionally, an x-markdown-tokens estimate. Browsers keep getting HTML. Cloudflare can do this at the edge for zones that enable it; we generate a Markdown twin of each page at build time instead.

Who should implement it: content sites that want agents to read pages without parsing navigation and scripts. Who should not: sites whose pages are mostly app UI. Evidence: none found for citations. Google's guidance says Markdown is not needed to appear in its AI features.

curl -s -D - -o /dev/null -H 'Accept: text/markdown' https://example.com/
# expect: content-type: text/markdown; vary: Accept
curl -s -D - -o /dev/null https://example.com/
# expect: content-type: text/html

Bot access control: AI crawler rules in robots.txt

What it is: named robots.txt groups for AI crawlers, so search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) can get different rules. A * group alone also passes but applies one policy to everyone.

Who should implement it: any site with a view on AI training versus AI search. Who should not: nobody, but check each vendor's crawler page for current names. User-triggered fetchers such as ChatGPT-User may not apply robots.txt. Evidence: as robots.txt. Our free AI crawler access checker reads any site's rules per crawler.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Test
curl -s https://citerlabs.com/api/crawler-check -H 'Content-Type: application/json' -d '{"url":"example.com"}'

Bot access control: Content Signals

What it is: a Content-Signal line in robots.txt stating whether you allow search, ai-input (use in AI answers, such as retrieval) and ai-train. Defined by Cloudflare's Content Signals Policy and an IETF draft from the AI Preferences work. Cloudflare's managed robots.txt adds a comment block explaining the signals; copy it if you write your own.

Who should implement it: sites that want to state a preference. Who should not: nobody, but know it is a stated preference, not a block. Put it in every named group too, because a named group replaces *. Evidence: none found that it changes AI behaviour or visibility.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

# Test
curl -s https://example.com/robots.txt | grep -i content-signal

Bot access control: Web Bot Auth

What it is: your bot signs its HTTP requests (HTTP Message Signatures, RFC 9421, with an Ed25519 key) and you publish the public keys as a JWKS at /.well-known/http-message-signatures-directory with the media type application/http-message-signatures-directory+json. Sites can then verify the request really came from you. Spec: the IETF webbotauth working group draft; Cloudflare verifies it.

Who should implement it: only if your site sends bot or agent requests to other sites. Who should not: a site that sends no bot traffic gains nothing; the scanner reports this check as informational. We publish one because our crawler access checker fetches other sites. Evidence for visibility: none; this is about identifying your bot.

A detail that matters today: the working group draft asks signers to send Signature-Agent as a dictionary (sig1="https://..."), while Cloudflare's verifier documentation says it rejects that form and wants the older quoted-string form. Check your verifiers before choosing.

GET /.well-known/http-message-signatures-directory
Content-Type: application/http-message-signatures-directory+json
{"keys":[{"kty":"OKP","crv":"Ed25519","kid":"<JWK thumbprint>","x":"<public key>","use":"sig"}]}

# On each request your bot sends:
Signature-Agent: "https://example.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1793232000;keyid="<thumbprint>";alg="ed25519";expires=1793232060;nonce="...";tag="web-bot-auth"
Signature: sig1=:<base64 signature>:

# Test
curl -sI https://example.com/.well-known/http-message-signatures-directory

Discovery: API catalog (RFC 9727)

What it is: a JSON linkset at /.well-known/api-catalog, served as application/linkset+json, listing your public APIs with links to their OpenAPI description (service-desc), human docs (service-doc) and a status endpoint.

Who should implement it: sites with a public API. Who should not: sites without one; do not list form endpoints meant only for people. Evidence: none found for AI visibility.

{
  "linkset": [{
    "anchor": "https://example.com/api/v1",
    "service-desc": [{ "href": "https://example.com/openapi.json", "type": "application/vnd.oai.openapi+json" }],
    "service-doc": [{ "href": "https://example.com/docs", "type": "text/html" }],
    "status": [{ "href": "https://example.com/api/v1/health" }]
  }]
}

// Test: curl -sI https://example.com/.well-known/api-catalog  (expect application/linkset+json)

Discovery: OAuth / OpenID Connect discovery

What it is: authorization server metadata at /.well-known/oauth-authorization-server (RFC 8414) or /.well-known/openid-configuration, listing issuer, authorization and token endpoints and supported grants, so agents can sign in.

Who should implement it: only sites that run a real OAuth or OIDC server. Who should not: everyone else. Publishing it without a working server sends agents into a broken sign-in. Evidence: none found for visibility.

curl -s https://example.com/.well-known/oauth-authorization-server | jq '.issuer, .token_endpoint'

Discovery: OAuth protected resource metadata (RFC 9728)

What it is: /.well-known/oauth-protected-resource tells a client which authorization server protects your API and which scopes exist. A 401 response can point to it with WWW-Authenticate: Bearer resource_metadata="...". Remote MCP servers that need sign-in use this.

Who should implement it: APIs and MCP servers that require OAuth tokens. Who should not: public APIs with no authentication. Evidence: none found for visibility.

{
  "resource": "https://api.example.com",
  "authorization_servers": ["https://auth.example.com"],
  "scopes_supported": ["read"],
  "bearer_methods_supported": ["header"]
}

// Test: curl -s https://example.com/.well-known/oauth-protected-resource

Discovery: auth.md

What it is: a Markdown file at /auth.md, with an H1 containing "auth.md", that tells agents how to register or get credentials. With OAuth it points at your metadata; without it, it describes the registration endpoint and methods.

Who should implement it: services where agents can register or be provisioned. Who should not invent a registration flow just to pass: our /auth.md says plainly that no registration exists and all our endpoints are public, which the scanner reports as "does not describe agent registration". That is the correct answer for us. Evidence: none found for visibility.

curl -s https://example.com/auth.md | head -5

Discovery: MCP server card

What it is: a JSON file at /.well-known/mcp/server-card.json describing your remote MCP server: name, version, transport endpoint and capabilities. The format is still a proposal (SEP-2127, an experimental extension), so include both the newer fields (name, version, description, remotes) and the ones scanners check (serverInfo, transport endpoint, capabilities).

Who should implement it: only if the MCP server works. Test initialize and tools/list before publishing the card. Who should not: anyone without a running server. Evidence: none found for visibility.

curl -s https://example.com/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"curl","version":"1"}}}'
curl -s https://example.com/mcp -H 'Content-Type: application/json' -H 'MCP-Protocol-Version: 2025-11-25' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}'
curl -s https://example.com/.well-known/mcp/server-card.json

Discovery: A2A agent card

What it is: /.well-known/agent-card.json describes an agent that other agents can call over the Agent2Agent protocol: name, version, interfaces (endpoint and binding), capabilities and skills.

Who should implement it: only if you run a working A2A agent. Who should not: a site without one. Keep documentationUrl pointing at a page that exists; ours currently points at a page that returns 404, which we have flagged for a fix. Evidence: none found for visibility.

curl -s https://example.com/.well-known/agent-card.json | jq '.name, .supportedInterfaces'

Discovery: Agent Skills index

What it is: /.well-known/agent-skills/index.json (Agent Skills Discovery v0.2.0) lists SKILL.md files that teach an agent how to use your product. Each entry has name, type (skill-md or archive), description, url and digest, where digest is sha256: plus the hex SHA-256 of the file's exact bytes. Generate the digest at build time so it cannot go stale.

Who should implement it: products with tasks an agent can do with your API or content. Who should not: sites with nothing for an agent to do. Evidence: none found for visibility.

{
  "$schema": "https://schemas.agentskills.io/discovery/0.2.0/schema.json",
  "skills": [{
    "name": "check-ai-crawler-access",
    "type": "skill-md",
    "description": "Check which AI crawlers a site's robots.txt allows.",
    "url": "/.well-known/agent-skills/check-ai-crawler-access/SKILL.md",
    "digest": "sha256:<64 hex characters>"
  }]
}

// Test: curl -s https://example.com/.well-known/agent-skills/<name>/SKILL.md | shasum -a 256

Discovery: WebMCP

What it is: a browser API (a W3C community group draft) that lets a page register tools an in-browser AI agent can call. The current draft puts it on document.modelContext; older Chrome builds used navigator.modelContext. registerTool returns a promise, and aborting the signal you pass unregisters the tool.

Who should implement it: pages with actions an agent could do for a user (search, a calculator, a free tool). Who should not: pages with nothing to call. Feature-detect so nothing breaks in browsers without it. Evidence: none found for visibility.

const mc = document.modelContext ?? navigator.modelContext;
if (mc?.registerTool) {
  const controller = new AbortController();
  mc.registerTool({
    name: "search_docs",
    description: "Search this site's documentation.",
    inputSchema: { type: "object", properties: { q: { type: "string" } }, required: ["q"] },
    execute: async ({ q }, { signal } = {}) => (await fetch("/search?q=" + encodeURIComponent(q), { signal })).text(),
  }, { signal: controller.signal }).catch(() => {});
}

Discovery: ARD capability manifest (AI catalog)

What it is: /.well-known/ai-catalog.json lists your agent-facing artifacts (MCP server cards, A2A agent cards, skills, APIs) using the AI Catalog data model, with urn:air identifiers and a few representative queries per entry. Both the ARD spec and the AI Catalog spec are drafts.

Who should implement it: sites with at least one real artifact to list. Who should not: sites with nothing to list. Evidence: none found for visibility.

{
  "specVersion": "1.0",
  "host": { "displayName": "Example", "identifier": "did:web:example.com" },
  "entries": [{
    "identifier": "urn:air:example.com:mcp:main",
    "displayName": "Example MCP server",
    "type": "application/mcp-server-card+json",
    "url": "https://example.com/.well-known/mcp/server-card.json",
    "representativeQueries": ["check stock for a product", "track an order"]
  }]
}

// Test: curl -s https://example.com/.well-known/ai-catalog.json | jq '.entries | length'

Discovery: Personal Agent Protocol

What it is: a draft (0.1) for letting a user's personal agent sign in and act on their account, discovered through /.well-known/poppy.json. Declaring APIs, agents or browser sessions requires an OAuth issuer.

Who should implement it: companies whose signed-in users would let an agent act for them, with a real OAuth issuer. Who should not: sites without accounts for the public. A website-only declaration would say only that your site can be browsed. Evidence: none; the scanner treats it as informational.

Commerce: x402, MPP, UCP, ACP and AP2

What they are: protocols for agents to pay. x402 returns HTTP 402 with payment requirements on paid routes; MPP adds x-payment-info to OpenAPI operations; UCP publishes /.well-known/ucp; ACP publishes /.well-known/acp.json; AP2 is a payments extension declared in an A2A agent card.

Who should implement them: shops and paid APIs that want agents to buy. Who should not: everyone else; the scanner marks them neutral, not failed, for non-commerce sites and does not score them. Evidence: none for visibility.

What about llms.txt?

The scanner does not score llms.txt, but its fix list includes it. It is a community proposal, not a standard; see our llms.txt guide for what is known. We publish one and point to it with a describedby Link header.

What we deliberately did not implement on citerlabs.com, and why

• OAuth discovery and OAuth protected resource metadata: we have no OAuth server and no OAuth-protected API. Publishing the metadata would send agents into a sign-in that does not exist. If we ever add an authenticated MCP tier with higher limits, it would need a real login, an authorization server and client registration first.
• Agent registration in auth.md: there is nothing to register for. Our auth.md says so, lists the public endpoints and limits, and tells agents not to create accounts in the client portal.
• Personal Agent Protocol: our only signed-in area is a client portal for people, so there is no account an agent could act on.
• DNS-AID: it needs SVCB records and DNSSEC on the DNS zone. The records for our A2A endpoint and MCP server are written down for the domain owner to add; we did not change DNS.
• Commerce protocols: we do not sell anything an agent can buy online.

How to test your own site

Then check by hand anything the scanner cannot see: whether your CDN or firewall blocks AI crawlers, and whether the services your metadata points to actually work. Download the CSV below to track each check.

curl -s -X POST https://isitagentready.com/api/scan -H 'content-type: application/json' -d '{"url":"https://example.com"}' \
  | jq '.checks | to_entries[] | .value | to_entries[] | "\(.key): \(.value.status)"'