---
title: "AI Crawler Access Checker | CiterLabs"
url: "https://citerlabs.com/tools/crawler-access-check"
description: "Free check of what GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 12 other crawlers may fetch on your site, read from your robots.txt. No sign-up."
last_updated: "2026-10-28"
---

# Which AI crawlerscan read your site?

Enter a domain. We read its robots.txt and show, for 16 AI and search crawlers, whether the homepage is allowed and which rule decided it. We also show the HTTP status our own checker received.

Run the check

Free, no sign-up, 20 checks per day. Results are cached for an hour. We fetch only `/robots.txt` and the homepage.

## How this check works

We fetch `https://your-domain/robots.txt` and apply RFC 9309: the most specific matching rule wins, and a crawler named in its own group ignores the `User-agent: *` group. A 4xx robots.txt means crawlers may fetch everything; a 5xx means they should assume nothing is allowed.

Our checker identifies itself as `CiterLabsCrawlerCheck/1.0` and signs its requests with Web Bot Auth (HTTP Message Signatures) when the key is configured. Our public keys are at [/.well-known/http-message-signatures-directory](https://citerlabs.com/.well-known/http-message-signatures-directory). We do not pretend to be GPTBot or any other crawler, so a site that filters by user agent or IP address may answer them differently.

robots.txt is a request, not a lock. Firewalls and bot-management settings can block crawlers that robots.txt allows. Check your CDN settings too. For the wider picture, see the [AI crawler access guide](https://citerlabs.com/resources/ai-crawler-access-guide) and the [agent readiness checklist](https://citerlabs.com/resources/agent-readiness-checklist) .

Agents can call this check directly: `POST /api/crawler-check` (see [openapi.json](https://citerlabs.com/openapi.json)) or the MCP tool `check_ai_crawler_access` at `/mcp`.