# Robots.txt checker and tester, with sitemap check

> Test whether Googlebot, Bingbot and AI crawlers like GPTBot can read a page under your robots.txt, and check that your XML sitemap is found and valid.

Canonical: https://indielaunch.dev/tools/robots-txt-checker
Last updated: 2026-11-03

Paste your site's address to read its robots.txt and sitemap. The checker tests whether Google, Bing and the main AI crawlers are allowed to read a page, using the same matching rules Google documents, and checks that your sitemap loads and lists your pages.

The tool itself is interactive: open [Robots.txt & sitemap checker](https://indielaunch.dev/tools/robots-txt-checker) in a browser to use it.

## How to use it

1. Paste your site's address. To test a particular page, paste that page's address instead.
2. Press Check. We read /robots.txt, then each sitemap it lists (or /sitemap.xml if it lists none).
3. Look at the crawler table for anything blocked that shouldn't be, then at the sitemap notes.

## What it checks

- Whether robots.txt exists and loads, and what its status code means to Google.
- Allowed or blocked for Googlebot, Bingbot, Applebot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot, with the rule that decided it.
- Each sitemap's status, whether it's a sitemap or a sitemap index, and how many addresses it lists.
- Common mistakes, like blocking the whole site or not pointing to a sitemap.

## FAQ

### What does robots.txt do?

robots.txt is a plain text file at the root of a site that tells crawlers which paths they may read. It controls crawling, not indexing: a page blocked in robots.txt can still appear in Google if other sites link to it. To keep a page out of results, let it be crawled and give it a noindex tag instead.

### How does Google decide which rule wins?

Google uses the group for the most specific user agent that matches (Googlebot's own group over the * group), then the longest matching rule in it. If an Allow and a Disallow rule match with the same length, Allow wins. Paths are case-sensitive, and * and $ work as wildcards.

### Should I block AI crawlers like GPTBot?

It's a trade-off. Blocking them keeps your content out of some AI training sets, but it can also keep your product out of AI answers that people now use to find tools. Google-Extended only controls use in Google's AI models; it doesn't affect Google Search.

### What if my robots.txt returns an error?

Google treats a missing robots.txt (404) as permission to crawl everything. A server error (5xx) is different: Google may stop crawling the site until the file loads again, so a broken robots.txt can hurt more than having none.

## Related

- [Meta title & description checker](https://indielaunch.dev/tools/meta-tag-checker)
- [Open Graph checker](https://indielaunch.dev/tools/open-graph-checker)
- [Dofollow / nofollow link checker](https://indielaunch.dev/tools/nofollow-link-checker)
- [Submit your startup](https://indielaunch.dev/submit-your-startup)