Skip to content

    Which AI crawlerscan read your site?

    Enter a domain. We read its robots.txt and show, for 16 AI and search crawlers, whether the homepage is allowed and which rule decided it. We also show the HTTP status our own checker received.

    Run the check

    Free, no sign-up, 20 checks per day. Results are cached for an hour. We fetch only /robots.txt and the homepage.

    How this check works

    We fetch https://your-domain/robots.txt and apply RFC 9309: the most specific matching rule wins, and a crawler named in its own group ignores the User-agent: * group. A 4xx robots.txt means crawlers may fetch everything; a 5xx means they should assume nothing is allowed.

    Our checker identifies itself as CiterLabsCrawlerCheck/1.0 and signs its requests with Web Bot Auth (HTTP Message Signatures) when the key is configured. Our public keys are at /.well-known/http-message-signatures-directory. We do not pretend to be GPTBot or any other crawler, so a site that filters by user agent or IP address may answer them differently.

    robots.txt is a request, not a lock. Firewalls and bot-management settings can block crawlers that robots.txt allows. Check your CDN settings too. For the wider picture, see the AI crawler access guide and the agent readiness checklist .

    Agents can call this check directly: POST /api/crawler-check (see openapi.json) or the MCP tool check_ai_crawler_access at /mcp.