How to use it
- Paste your site's address. To test a particular page, paste that page's address instead.
- Press Check. We read /robots.txt, then each sitemap it lists (or /sitemap.xml if it lists none).
- Look at the crawler table for anything blocked that shouldn't be, then at the sitemap notes.
What it checks
- Whether robots.txt exists and loads, and what its status code means to Google.
- Allowed or blocked for Googlebot, Bingbot, Applebot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot, with the rule that decided it.
- Each sitemap's status, whether it's a sitemap or a sitemap index, and how many addresses it lists.
- Common mistakes, like blocking the whole site or not pointing to a sitemap.
Free, with no sign-up. We read the page once to answer and don't store what you check.
Frequently asked questions
- What does robots.txt do?
- robots.txt is a plain text file at the root of a site that tells crawlers which paths they may read. It controls crawling, not indexing: a page blocked in robots.txt can still appear in Google if other sites link to it. To keep a page out of results, let it be crawled and give it a noindex tag instead.
- How does Google decide which rule wins?
- Google uses the group for the most specific user agent that matches (Googlebot's own group over the * group), then the longest matching rule in it. If an Allow and a Disallow rule match with the same length, Allow wins. Paths are case-sensitive, and * and $ work as wildcards.
- Should I block AI crawlers like GPTBot?
- It's a trade-off. Blocking them keeps your content out of some AI training sets, but it can also keep your product out of AI answers that people now use to find tools. Google-Extended only controls use in Google's AI models; it doesn't affect Google Search.
- What if my robots.txt returns an error?
- Google treats a missing robots.txt (404) as permission to crawl everything. A server error (5xx) is different: Google may stop crawling the site until the file loads again, so a broken robots.txt can hurt more than having none.
Launch your product on IndieLaunch
Free to list, with a product page that search engines and AI answers can find, and a dofollow link to your site with Premium or our badge.